Kibana Loading Sample data

In this section, we are going to learn how to load sample data in the Kibana. Alongwith this, we will also learn about how to load data in Kibana using the Dev tool, load data using the Elasticsearch and Logstash, using dev tool to insert the bulk data.

We’ve seen how to upload the data to Elasticsearch from Logstash. In our Kibana interface, we are going to upload the data with the help of the Logstash and the Elasticsearch. Later in this section, we will hear about the data that has date, longitude, and latitude fields that we need to use. Also, if we don’t have a CSV file, we will see how to upload data directly to Kibana.

Use of the Logstash upload method for Elasticsearch field data in Kibana

The data that we are going to use will be in the form of the CSV file format. We have taken the data from the popular data repository website Kaggle.com. From here, many of the researchers and the analyst take data to form the research purpose.

The medical visits to use the data home here were picked up from the Kaggle.com website.

The following are the CSV-file fields available:

  [“Visit_Status”,”Time_Delay”,”City”,”City_id”,”Patient_Age”,”Zipcode”,”Latitude”,”Longitude”,  “Pathology”,”Visiting_Date”,”Id_type”,”Id_personal”,”Number_Home_Visits”,”Is_Patient_Minor”,”Geo_point”]  

The Home visits.csv is like this:

Kibana Loading Sample data

The following is the Logstash conf file to be used:

Kibana Loading Sample data

By default, in Elasticsearch, Logstash regards everything to be uploaded as a string. In case the date field of the CSV file that we need does not have the desired data format, then one can perform the following given code to get the desired date format.

For date field:

Kibana Loading Sample data

In the case of geo location, Elasticsearch has the same understanding as:

Kibana Loading Sample data

And we need to make sure that we need it in the same format of Elasticsearch for Longitude as well as for Latitude. So, for that first, we need to transform the longitude and latitude into the float data type.

Now we must rename it so that it can be made usable as part of the lat and lon position json set. See the code below for the reference:

Kibana Loading Sample data

Using the following code for converting fields to integers:

Kibana Loading Sample data

After you have taken care of the fields, run the following command to upload data to Elasticsearch:

Go to the Logstash bin directory, and run the command below.

When done, you can see the index in Elasticsearch, as shown below in the Logstash conf file.

Kibana Loading Sample data

Now we can create an index pattern on the index uploaded above and use it to create visualization further.

Using Dev Tools to Upload Bulk Data

We will use Kibana UI Dev Tools. The Dev Tools in the Kibana is very helpful in uploading data into the Elasticsearch, even without using Logstash in our Kibana. With the help of the dev tool in the Kibana, we can post, delete, put, and also search the data in the Kibana.

Let’s take the JSON data from the URL below and upload the same in Kibana. In the same way, you can try loading some sample json data inside Kibana.

We can use the following code for our reference:

  [     {“type”:”act”,”line_id”:1,”play_name”:”Henry IV”,           “speech_number”:””,”line_number”:””,”speaker”:””,”text_entry”:”ACT I”},     {“type”:”scene”,”line_id”:2,”play_name”:”Henry IV”,     “speech_number”:””,”line_number”:””,”speaker”:””,”text_entry”:”SCENE I.London. The palace.”},     {“type”:”line”,”line_id”:3,”play_name”:”Henry IV”,     “speech_number”:””,”line_number”:””,”speaker”:””,”text_entry”:     “Enter KING HENRY, LORD JOHN OF LANCASTER, the      EARL of WESTMORELAND, SIR WALTER BLUNT, and others”}  ]  

The json code to be used with Kibana has to be with indexed as follows:

  {“index”:{“_index”:”shakespeare”,”_id”:0}}  {“type”:”act”,”line_id”:1,”play_name”:”Henry IV”,   “speech_number”:””,”line_number”:””,”speaker”:””,”text_entry”:”ACT I”}  {“index”:{“_index”:”shakespeare”,”_id”:1}}  {“type”:”scene”,”line_id”:2,”play_name”:”Henry IV”,  “speech_number”:””,”line_number”:””,”speaker”:””,  “text_entry”:”SCENE I. London. The palace.”}  {“index”:{“_index”:”shakespeare”,”_id”:2}}  {“type”:”line”,”line_id”:3,”play_name”:”Henry IV”,  “speech_number”:””,”line_number”:””,”speaker”:””,”text_entry”:  “Enter KING HENRY, LORD JOHN OF LANCASTER, the EARL   of WESTMORELAND, SIR WALTER BLUNT, and others”}  

Note that there is an additional data that goes in the JSON file that is:

To convert any Elasticsearch-compatible sample JSON file, we have a small PHP code that will output the JSON file to the format that Elasticsearch requires.

PHP Code

Kibana Loading Sample data