4.1. Quality evaluation
4.1.1 Accuracy and Fluency
In this part, the source text and the subtitle generated by CC subtitling group (CCS) are used as the reference for the quality evaluation. Comparing MTS-B, MTS-Y with the source text the CCS, the translator calculated the number of lines with errors in terms of accuracy (addition, mistranslation and omission) and fluency (grammar and spelling), which was shown in Table 1. There are 47 lines in MTS-B while 46 lines in MTS-Y. For example, if there are grammar mistakes in line 1, the number of lines in Grammar will be 1. If all lines in MTS-B have mistranslation, the number in Mistranslation will be 47. Table 1 shows that both MTS-B and MTS-Y have no spelling. There are less errors in terms of accuracy and fluency in MTS-Y than in MTS-B.
Table 1. The number of lines with errors
|
MTS-B
|
MTS-Y
|
Accuracy
|
Addition
|
0
|
1
|
Mistranslation
|
12
|
9
|
Omission
|
2
|
1
|
Fluency
|
Grammar
|
4
|
3
|
Spelling
|
0
|
0
|
Total
|
18
|
13
|
For typography, the author compared MTS-B with MTS-Y mainly in the number of lines and position on screen, punctuation usage, and subtitle type. Figure 3 shows some examples of MTS-B and MTS-Y. For the number of lines, there is only one line per subtitle in MTS-B while two lines per subtitle in MTS-Y. Besides, both MTS-B and MTS-Y are placed at the bottom of the screen. However, subtitles are placed at the center on Bilibili while on YouTube, subtitles are on the left. MTS-B seldom use punctuation, only including three commas and one question marks. However, MTS-Y uses much more punctuation, including 27 commas, which seems less natural. Furthermore, MTS-B are edited subtitles while MTS-Y are verbatim subtitles. All these aspects show that MTS-B is cleaner and simpler than MTS-Y.
4.1.2 Design, Timing and Style
The segment length (the number of characters per line) and the characters per second (CPS) were calculated by the audiovisual translation software-ysj(人人译视界)7. In this study, a one-way ANOVA (Analysis of Variance) was carried out to test the difference of the segment length and the CPS between MTS-B and MTS-Y. In this case, the condition is set that α= 0.05 and the results were calculated by Excel (see Table 2).
Table 2 The segment length and CPS of MTS-B and MTS-Y
|
Max
|
Min
|
Mean
|
Variance
|
P-value
|
Length
|
MTS-B
|
24
|
1
|
9.79
|
34.91
|
0.28
|
MTS-Y
|
30
|
5
|
10.93
|
17.53
|
CPS
|
MTS-B
|
8
|
1
|
4.11
|
2.53
|
0.24
|
MTS-Y
|
14
|
1
|
4.57
|
4.43
|
Although there is significant difference of variance between the segment length of MTS-B and MTS-Y, the p-value is much higher than 0.05, which shows no significant difference between the segment length of MTS-B and MTS-Y. Furthermore, there is also no significant difference between the CPS of MTS-B and MTS-Y.
When it comes to style, this study mainly analyzes the register. The video is talking about the introduction of world mythology crash course and the target readers are all online users who are interested in this course. Although the course is talking about history, the speaker uses personnal pronouns, such as ‘I’, ‘we’, and ‘you’and hopes to get closer to the audience. Therefore, the register of the video is semi-formal. Both subtitles on Bilibili and YouTube are automatically translated by machine and most of significant difference between MTS-B and MTS-Y.
To conclude, MTS-Y performs better in accuracy and the grammar and spelling of fluency. However, MTS-B shows cleaner and simpler subtitles than MTS-Y. There is no significant difference between MTS-B and MTS-Y, in terms of design, timing and style. Therefore, hypothesis 1 is not all correct. All the aspects mentioned above will be considered in the following part to find out the relationship between quality and the reception of the audience.
4.2 Demographic profile
The study collects responses from Group MTS-B and Group MTS-Y. There are 22 participants in each group. A total of 44 questionnaires were completed and valid, so the response rate is 100%. Table 3 shows the demographics of two groups, including gender, age, English proficiency, the education level and the experience of translating subtitles.
Table 3 Demographics of participants in Group MTS-B and Group MTS-Y
|
Group MTS-B (n=22)
|
Group MTS-Y (n=22)
|
Gender
|
Male
|
4
|
18.18%
|
8
|
36.36%
|
Female
|
18
|
81.82%
|
14
|
63.64%
|
Total
|
22
|
100%
|
22
|
100%
|
Age
|
18-24
|
8
|
36.36%
|
11
|
50%
|
25-34
|
14
|
63.64%
|
8
|
36.36%
|
35-44
|
0
|
0
|
3
|
13.64%
|
Total
|
22
|
100%
|
22
|
100%
|
English Proficiency
|
Level A
|
8
|
36.36%
|
8
|
36.36%
|
Level B
|
9
|
40.91%
|
9
|
40.91%
|
Level C
|
5
|
22.73%
|
5
|
22.73%
|
Total
|
22
|
100%
|
22
|
100%
|
Highest education degree earned
|
High School Diploma
|
1
|
4.54%
|
0
|
0
|
Bachelor’s or College Degree
|
14
|
63.64%
|
14
|
63.64%
|
Master’s Degree
|
7
|
31.82%
|
8
|
36.36%
|
Total
|
22
|
100%
|
22
|
100%
|
Experience of Translating Subtitles
|
No
|
19
|
86.36%
|
19
|
86.36%
|
Yes
|
3
|
13.64%
|
3
|
13.64%
|
Total
|
22
|
100%
|
22
|
100%
|
In this study, it is surprising that the information of English proficiency and the experience of translating subtitles in two groups are exactly the same, although the participants are assigned randomly. For two groups, most of the participants don’t have any experience of translating subtitles, which shows that they can give comments as the general audience instead of professions. There are other similarities between two groups. For example, males are much less than females and the majority of the participants got Bachelor or College Degree. However, participants aged 18-24 in Group MTS-B are less than those in Group MTS-Y, while participants aged 25-34 in Group MTS-B are more than those in Group MTS-Y.
4.3 Comprehension testing
The video is just the review of crash course world mythology (1’58), so there are only two questions in this part. The comprehension questions aim to test the participants’ understanding of the key points of the video, which contain conjunctions. Here are these two questions:
6. This crash course is going to address which of the following questions:
A. where children come from
B. how we got death and disease
C. what the deal is with unicorns and virgins
D. all of the above
7. What is the key point of this crash course?
A. tick all of the mythological locales off of the geographical list one by one
B. see the similarities in the stories that different people tell about the universe
The results are shown on Figure 4. In question 6, D is the correct answer and presents a parallel relation in the video. In question 7, B is the correct answer and stressed in the video using ‘though’. For the comprehension questions, Group MTS-B shows the similarity in comprehension testing with Group MTS-Y. For the quality mentioned above, MTS-Y performs better than MTS-B in terms of accuracy and grammar and spelling of fluency. For typography, MTS-B shows cleaner and simpler subtitles than MTS-Y. However, it seems that the accuracy and the fluency of the subtitles have little impact on the comprehension of the audience, which goes against the hypothesis 2. Moreover, demographics of two groups show differences in gender and age, but it seems that they don’t affect the comprehension of the audience.
This part consists of 7 five-point Likert scale statements as well as one open question. Table 4 shows the distribution of different answers from Likert scale statements for Group MTS-B and Group MTS-Y in percentage form (absolute values are also shown in parentheses). For instance, if all the participants in Group MTS-B chose ‘Strongly agree’ for all the 7 statements, then this scale would have been chosen 154 times (22 ×7). However, it was chosen 42 times, less than one third, by Group MTS-B.
Table 4. Attitude survey results from Group MTS-B and Group MTS-Y
|
Strongly agree
|
Agree
|
Neutral
|
Disagree
|
Strongly disagree
|
Group MTS-B
|
27.27% (42)
|
39.61% (61)
|
28.57% (44)
|
4.55% (7)
|
0
|
Group MTS-Y
|
34.42% (53)
|
43.51% (67)
|
18.18% (28)
|
3.89% (6)
|
0
|
According to Table 4, for Group MTS-B and Group MTS-Y, over half of the participants chose ‘Strongly agree’ and ‘agree’, while no on chose ‘Strongly disagree’, indicating that participants’ attitudes to MTS-B and MTS-Y are mostly positive. In respect of the options ‘Strongly agree’ and ‘agree’, Group MTS-Y has higher number than Group MTS-B. It is surprising that in Group MTS-B, there are more participants who chose ‘neutral’ than those who chose ‘strongly agree’.
For each statement, the percentage of agreements per group was calculated. In Group MTS-B, nearly half of the participants chose ‘neutral’ for the statement 8, much more than those in Group MTS-Y who chose ‘neutral’, indicating that MTS-B are less helpful for participants to fully understand the contents of the crash course.
For the open question, each participant per group was asked to give comments to the subtitles they are received. Although participants in MTS-B express less agreements than those in MTS-Y, 7 of them showed no comments towards MTS-B, more than those participants in MTS-Y. Other participants show their opinions towards the video they are received. For MTS-B, 3 participants noticed that there are some errors in accuracy, while 3 participants advise that it is necessary to provide bilingual subtitle. Some of them noticed the grammar mistakes while others cared about the typography of the font. For MTS-Y, most participants cared about typography of the subtitle, for example, the position and verbatim subtitles are not comfortable for reading. 2 of them noticed that grammar mistakes and suggest the bilingual subtitle. However, none of the participants mentioned the accuracy.
It can be seen that participants had a better attitude towards MTS-Y than MTS-B and they don’t have comments on the accuracy of MTS-Y. From the previous results on quality evaluation, MTS-Y performs better in accuracy than MTS-B. Therefore, the result is consistent with hypothesis 3 that if the subtitle performs better in accuracy, the audience will have a better attitude towards it.
Although most participants said that the verbatim subtitles as well as their position in the screen are not comfortable, they show not significant differences in comprehension testing.
6 Website of Cambridge Assessment English: https://www.cambridgeenglish.org/test-your-english/general-english/
7 ysj software: achieved from https://www.1sj.tv/html/otherService/download.html?version=1665739518000