Sign in with Google
Question:
We have the following string which is a valid JSON written to a file on HDFS.
{
"id":"tag:search.twitter.com,2005:564407444843950080",
"objectType":"activity",
"actor":{
"objectType":"person",
"id":"id:twitter.com:2302910022",
"link":"http%3A%2F%2Fwww.twitter.com%2Fme7me4610012",
"displayName":"",
"postedTime":"2014-01-21T11:06:06.000Z",
"image":"https%3A%2F%2Fpbs.twimg.com%2Fprofile_images%2F563125491159162881%2FfypkHK3M_normal.jpeg",
"summary":"ضًـأّيِّعٌهّ أّروٌأّحًنِأّ تٌـشُـتٌـهّـيِّ مًنِ يِّفُـهّـمًهّـأّ فُـقُط حسابي بالإنستقرام lloooo_20",
"links":[
{
"href":null,
"rel":"me"
}
],
"friendsCount":10503,
"followersCount":10325,
"listedCount":12,
"statusesCount":84957,
"twitterTimeZone":null,
"verified":false,
"utcOffset":null,
"preferredUsername":"me7me4610012",
"languages":[
"ar"
],
"favoritesCount":17
},
"verb":"share",
"postedTime":"2015-02-08T12:56:35.000Z",
"generator":{
"displayName":"Twitter for Android",
"link":"http%3A%2F%2Ftwitter.com%2Fdownload%2Fandroid"
},
"provider":{
"objectType":"service",
"displayName":"Twitter",
"link":"http%3A%2F%2Fwww.twitter.com"
},
"link":"http%3A%2F%2Ftwitter.com%2Fme7me4610012%2Fstatuses%2F564407444843950080",
"body":"RT @sckud1: فيديو: إمام يرفض بغضب الصلاة على أحد قتلى حزب الله في سوريا بسبب إطلاق النار: ماعاد http%3A%2F%2Ft.co%2FC55SaQKmUV http%3A%2F%2Ft.co%2Ft5TjIln…",
"object":{
"id":"tag:search.twitter.com,2005:564407126526013440",
"objectType":"activity",
"actor":{
"objectType":"person",
"id":"id:twitter.com:462268717",
"link":"http%3A%2F%2Fwww.twitter.com/sckud1",
"displayName":"صفق الهوى",
"postedTime":"2012-01-12T19:24:17.000Z",
"image":"https%3A%2F%2Fpbs.twimg.com%2Fprofile_images%2F508424482885615616%2FmPBGZBPx_normal.jpeg",
"summary":"اعلانك في سوق الخليج يحقق لك الوصول الى اكثر من مليون متابع خليجي http%3A%2F%2Fmarketgulf.com",
"links":[
{
"href":"http%3A%2F%2Fmarketgulf.com",
"rel":"me"
}
],
"friendsCount":435237,
"followersCount":464951,
"listedCount":708,
"statusesCount":1071685,
"twitterTimeZone":"Riyadh",
"verified":false,
"utcOffset":"10800",
"preferredUsername":"sckud1",
"languages":[
"ar"
],
"location":{
"objectType":"place",
"displayName":"Made in K S A"
},
"favoritesCount":77
},
"verb":"post",
"postedTime":"2015-02-08T12:55:19.000Z",
"generator":{
"displayName":"Tweet Old Post",
"link":"http%3A%2F%2Fwww.ajaymatharu.com%2F"
},
"provider":{
"objectType":"service",
"displayName":"Twitter",
"link":"http%3A%2F%2Fwww.twitter.com"
},
"link":"http%3A%2F%2Ftwitter.com%2Fsckud1%2Fstatuses%2F564407126526013440",
"body":"فيديو: إمام يرفض بغضب الصلاة على أحد قتلى حزب الله في سوريا بسبب إطلاق النار: ماعاد http%3A%2F%2Ft.co%2FC55SaQKmUV http%3A%2F%2Ft.co%2Ft5TjIlnZgN",
"object":{
"objectType":"note",
"id":"object:search.twitter.com,2005:564407126526013440",
"summary":"فيديو: إمام يرفض بغضب الصلاة على أحد قتلى حزب الله في سوريا بسبب إطلاق النار: ماعاد http%3A%2F%2Ft.co%2FC55SaQKmUV http%3A%2F%2Ft.co%2Ft5TjIlnZgN",
"link":"http%3A%2F%2Ftwitter.com%2Fsckud1%2Fstatuses%2F564407126526013440",
"postedTime":"2015-02-08T12:55:19.000Z"
},
"favoritesCount":0,
"twitter_entities":{
"hashtags":[
],
"trends":[
],
"urls":[
{
"url":"http%3A%2F%2Ft.co%2FC55SaQKmUV",
"expanded_url":"http%3A%2F%2Fwww.hasterya.com%2Farchives%2F34688utm_source%3DReviveOldPost%26utm_medium%3Dsocial%26utm_campaign%3DReviveOldPost",
"display_url":"hasterya.com/archives/34688…",
"indices":[
85,
107
]
}
],
"user_mentions":[
],
"symbols":[
],
"media":[
{
"id":564407126341468160,
"id_str":"564407126341468160",
"indices":[
108,
130
],
"media_url":"http%3A%2F%2Fpbs.twimg.com%2Fmedia%2FB9UtSoJIQAA07-r.jpg",
"media_url_https":"https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FB9UtSoJIQAA07-r.jpg",
"url":"http%3A%2F%2Ft.co%2Ft5TjIlnZgN",
"display_url":"pic.twitter.com/t5TjIlnZgN",
"expanded_url":"http%3A%2F%2Ftwitter.com%2Fsckud1%2Fstatus%2F564407126526013440%2Fphoto%2F1",
"type":"photo",
"sizes":{
"large":{
"w":320,
"h":180,
"resize":"fit"
},
"thumb":{
"w":150,
"h":150,
"resize":"crop"
},
"small":{
"w":320,
"h":180,
"resize":"fit"
},
"medium":{
"w":320,
"h":180,
"resize":"fit"
}
}
}
]
},
"twitter_extended_entities":{
"media":[
{
"id":564407126341468160,
"id_str":"564407126341468160",
"indices":[
108,
130
],
"media_url":"http%3A%2F%2Fpbs.twimg.com%2Fmedia%2FB9UtSoJIQAA07-r.jpg",
"media_url_https":"https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FB9UtSoJIQAA07-r.jpg",
"url":"http%3A%2F%2Ft.co%2Ft5TjIlnZgN",
"display_url":"pic.twitter.com/t5TjIlnZgN",
"expanded_url":"http%3A%2F%2Ftwitter.com%2Fsckud1%2Fstatus%2F564407126526013440%2Fphoto%2F1",
"type":"photo",
"sizes":{
"large":{
"w":320,
"h":180,
"resize":"fit"
},
"thumb":{
"w":150,
"h":150,
"resize":"crop"
},
"small":{
"w":320,
"h":180,
"resize":"fit"
},
"medium":{
"w":320,
"h":180,
"resize":"fit"
}
}
}
]
},
"twitter_filter_level":"low",
"twitter_lang":"ar"
},
"favoritesCount":0,
"twitter_entities":{
"hashtags":[
],
"trends":[
],
"urls":[
{
"url":"http%3A%2F%2Ft.co%2FC55SaQKmUV",
"expanded_url":"http%3A%2F%2Fwww.hasterya.com%2Farchives%2F34688utm_source%3DReviveOldPost%26utm_medium%3Dsocial%26utm_campaign%3DReviveOldPost",
"display_url":"hasterya.com/archives/34688…",
"indices":[
97,
119
]
}
],
"user_mentions":[
{
"screen_name":"sckud1",
"name":"صفق الهوى",
"id":462268717,
"id_str":"462268717",
"indices":[
3,
10
]
}
],
"symbols":[
],
"media":[
{
"id":564407126341468160,
"id_str":"564407126341468160",
"indices":[
139,
140
],
"media_url":"http%3A%2F%2Fpbs.twimg.com%2Fmedia%2FB9UtSoJIQAA07-r.jpg",
"media_url_https":"https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FB9UtSoJIQAA07-r.jpg",
"url":"http%3A%2F%2Ft.co%2Ft5TjIlnZgN",
"display_url":"pic.twitter.com/t5TjIlnZgN",
"expanded_url":"http%3A%2F%2Ftwitter.com%2Fsckud1%2Fstatus%2F564407126526013440%2Fphoto%2F1",
"type":"photo",
"sizes":{
"large":{
"w":320,
"h":180,
"resize":"fit"
},
"thumb":{
"w":150,
"h":150,
"resize":"crop"
},
"small":{
"w":320,
"h":180,
"resize":"fit"
},
"medium":{
"w":320,
"h":180,
"resize":"fit"
}
},
"source_status_id":564407126526013440,
"source_status_id_str":"564407126526013440"
}
]
},
"twitter_extended_entities":{
"media":[
{
"id":564407126341468160,
"id_str":"564407126341468160",
"indices":[
139,
140
],
"media_url":"http%3A%2F%2Fpbs.twimg.com%2Fmedia%2FB9UtSoJIQAA07-r.jpg",
"media_url_https":"https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FB9UtSoJIQAA07-r.jpg",
"url":"http%3A%2F%2Ft.co%2Ft5TjIlnZgN",
"display_url":"pic.twitter.com/t5TjIlnZgN",
"expanded_url":"http%3A%2F%2Ftwitter.com%2Fsckud1%2Fstatus%2F564407126526013440%2Fphoto%2F1",
"type":"photo",
"sizes":{
"large":{
"w":320,
"h":180,
"resize":"fit"
},
"thumb":{
"w":150,
"h":150,
"resize":"crop"
},
"small":{
"w":320,
"h":180,
"resize":"fit"
},
"medium":{
"w":320,
"h":180,
"resize":"fit"
}
},
"source_status_id":564407126526013440,
"source_status_id_str":"564407126526013440"
}
]
},
"twitter_filter_level":"low",
"twitter_lang":"ar",
"retweetCount":1,
"gnip":{
"matching_rules":[
{
"tag":"ISIS66"
}
],
"urls":[
{
"url":"http%3A%2F%2Ft.co%2Ft5TjIlnZgN",
"expanded_url":"http%3A%2F%2Ftwitter.com%2Fsckud1%2Fstatus%2F564407126526013440%2Fphoto%2F1",
"expanded_status":200
},
{
"url":"http%3A%2F%2Ft.co%2FC55SaQKmUV",
"expanded_url":"http%3A%2F%2Fwww.hasterya.com%2Farchives%2F34688utm_source%3DReviveOldPost%26utm_medium%3Dsocial%26utm_campaign%3DReviveOldPost",
"expanded_status":200
}
],
"klout_score":50,
"language":{
"value":"ar"
}
}
}
EDIT
We configure a flume agent that reads the data from that file and pass it to Solr sink but unfortunately this exception in the title is throw.
and here is the stack trace
org.kitesdk.morphline.api.MorphlineRuntimeException: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'http': was expecting ('true', 'false' or 'null')
at [Source: java.io.ByteArrayInputStream@20d7aa52; line: 1, column: 9]
at org.kitesdk.morphline.stdio.AbstractParser.doProcess(AbstractParser.java:98)
at org.kitesdk.morphline.base.AbstractCommand.process(AbstractCommand.java:156)
at org.kitesdk.morphline.stdlib.TryRulesBuilder$TryRules.doProcess(TryRulesBuilder.java:120)
at org.kitesdk.morphline.base.AbstractCommand.process(AbstractCommand.java:156)
at org.kitesdk.morphline.base.AbstractCommand.doProcess(AbstractCommand.java:181)
at org.kitesdk.morphline.base.AbstractCommand.process(AbstractCommand.java:156)
at org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.process(MorphlineHandlerImpl.java:128)
at org.apache.flume.sink.solr.morphline.MorphlineSink.process(MorphlineSink.java:141)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:744)
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'http': was expecting ('true', 'false' or 'null')
at [Source: java.io.ByteArrayInputStream@20d7aa52; line: 1, column: 9]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1524)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:557)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3095)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2340)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:818)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:698)
at com.fasterxml.jackson.databind.MappingIterator.hasNextValue(MappingIterator.java:159)
at org.kitesdk.morphline.json.ReadJsonBuilder$ReadJson.doProcess(ReadJsonBuilder.java:109)
at org.kitesdk.morphline.stdio.AbstractParser.doProcess(AbstractParser.java:96)
... 10 moreAnswer:
- Get All Questions & Answer for CDP Generalist Exam (CDP-0011) and trainings.
- Get All Questions & Answer for CDP Administrator - Private Cloud Base Exam CDP-2001 and trainings.
- Get All Questions & Answer for CDP Data Developer Exam CDP-3001 and trainings.
This Question is from QuickTechie Cloudera CDP Certification Preparation Kit.
We have the following string which is a valid JSON ...
Clearly the JSON parser disagrees!
However, the exception says that the error is at "line 1: column 9", and there is no "http" token near the beginning of the JSON. So I suspect that the parser is trying to parse something different than this string when the error occurs.
You need to find what JSON is actually being parsed. Run the application within a debugger, set a breakpoint on the relevant constructor for JsonParseException ... then find out what is in the ByteArrayInputStream that it is attempting to parse.
It might be obvious, but make sure that you are sending to the parser URL object not a String containing www adress. This will not work:
ObjectMapper mapper = new ObjectMapper();
String www = "www.sample.pl";
Weather weather = mapper.readValue(www, Weather.class);
But this will:
ObjectMapper mapper = new ObjectMapper();
URL www = new URL("http://www.oracle.com/");
Weather weather = mapper.readValue(www, Weather.class);
- Get All Questions & Answer for CDP Generalist Exam (CDP-0011) and trainings.
- Get All Questions & Answer for CDP Administrator - Private Cloud Base Exam CDP-2001 and trainings.
- Get All Questions & Answer for CDP Data Developer Exam CDP-3001 and trainings.
This Question is from QuickTechie Cloudera CDP Certification Preparation Kit.
I faced this exception for a long time and was not able to pinpoint the problem. The exception says line 1 column 9. The mistake I did is to get the first line of the file which flume is processing.
Apache flume process the content of the file in patches. So, when flume throws this exception and says line 1, it means the first line in the current patch.
If your flume agent is configured to use batch size = 100, and (for example) the file contains 400 lines, this means the exception is thrown in one of the following lines 1, 101, 201,301.
How to discover the line which causes the problem?
You have three ways to do that.
1- pull the source code and run the agent in debug mode. If you are an average developer like me and do not know how to make this, check the other two options.
2- Try to split the file based on the batch size and run the flume agent again. If you split the file into 4 files, and the invalid json exists between lines 301 and 400, the flume agent will process the first 3 files and stop at the fourth file. Take the fourth file and again split it into more smaller files. continue the process until you reach a file with only one line and flume fails while processing it.
3- Reduce the batch size of the flume agent to only one and compare the number of processed events in the output of the sink you are using. For example, in my case I am using Solr sink. The file contains 400 lines. The flume agent is configured with batch size=100. When I run the flume agent, it fails at some point and throw that exception. At this point check how many documents are ingested in Solr. If the invalid json exists at line 346, the number of documents indexed into Solr will be 345, so the next line is the line which causes the problem.
In my case I followed the third option and fortunately I pinpoint the line which causes the problem.
This is a long answer but it actually does not solve the exception. How I overcome this exception?
I have no idea why Jackson library complain while parsing a json string contains escaped characters \n \r \t. I think (but I am not sure) the Jackson parser is by default escaping these characters which cases the json string to be split into two lines (in case of \n) and then it deals each line as a separate json string.
In my case we used a customized interceptor to remove these characters before being processed by the flume agent. This is the way we solved this problem.
- Get All Questions & Answer for CDP Generalist Exam (CDP-0011) and trainings.
- Get All Questions & Answer for CDP Administrator - Private Cloud Base Exam CDP-2001 and trainings.
- Get All Questions & Answer for CDP Data Developer Exam CDP-3001 and trainings.
This Question is from QuickTechie Cloudera CDP Certification Preparation Kit.
.jpg)