- slide 1 of 23, currently active
- slide 2 of 23
- slide 3 of 23
- slide 4 of 23
- slide 5 of 23
- slide 6 of 23
- slide 7 of 23
- slide 8 of 23
- slide 9 of 23
- slide 10 of 23
- slide 11 of 23
- slide 12 of 23
- slide 13 of 23
- slide 14 of 23
- slide 15 of 23
- slide 16 of 23
- slide 17 of 23
- slide 18 of 23
- slide 19 of 23
- slide 20 of 23
- slide 21 of 23
- slide 22 of 23
- slide 23 of 23
마찬가지로 content도 띄어쓰기를 기준으로 토큰을 나눈다.
tokened_content = tokenizer.tokenize(sample_content) tokened_content[:20]
실행 결과
['현정부에서', '정규직', '일자리를', '늘리는', '것에', '찬성합니다', '.', '그런데', '공공기관', '비정규직들은', '인맥으로', '들어온', '경우가', '많습니다', '.', '자질이', '안되는데도', '정규직이', '된다면', '그']