Let’s build a real-time weather dashboard application with Apache Pinot and Apache Pulsar.
Join the DZone community and get the full member experience.
It is so easy to build Pulsar to Pinot applications for real-time analytics. I added another source of data for weather feeds for the U.S. I am looking at adding transit and aircraft data feeds next. The sky is the limit with Pulsar + Pinot. I will probably pull more sources from my big list from the Let’s Monitor talk.
Apache NiFi acquires our weather feed for the United States from NOAA. This is really easy to do, and I have it well-documented at the the source referenced.
Reference: https://github.com/tspannhw/SmartWeather
Reference: Weather How-To
The first thing we will need to do for infrastructure—put this in your DevOps box—is to create topics. We can let the producer of the first message automatically do this if you are on your laptop. Most production clusters want this pre-defined and run by an ops person. So here you go; we will take a look at the list of existing topics and then build it. Since Apache Pulsar is multi-tenant, you can create and specify a custom tenant and namespace for the weather application if you desire. Each organization will decide how to set up and build their hierarchy of tenants and namespaces that matches their application architecture landscape.
As part of the general application, after the data is sent from Apache NiFi to Apache Pulsar via the NiFi Connector, I have a Java Pulsar Function that creates a new schema for it.
