Replies: 1 comment 6 replies
-
YES!
Mpool surveillance in the first place
we'd build something anyways - with or without lily - piggybacking might be easier/cooler
I think the other end of the pipeline - the output part - is equally important. We would love to get data directly into an influxdb, for example. |
Beta Was this translation helpful? Give feedback.
-
|
no. i didn't know. looks pretty much like what we use now :D what options are there to write the CSV to right now? |
Beta Was this translation helpful? Give feedback.
-
I can't think of what you're asking here other than "What can I import these CSVs into?" but that seems like something you'd know. Can you rephrase? But if that is your question, CSVs can be imported into all sorts of shapes and postgres can import CSVs directly. (But you almost certainly know this.) Worst case, you can pick up some SaaS like fivetran to fit the CSV into whatever shape you need on the other end. And I quickly found this article which explains a few more solutions native to Influx toolchain. |
Beta Was this translation helpful? Give feedback.
-
|
PS: Maybe you're asking when are we going to export CSVs to current heights? This is work currently in-progress which we are blocked on due the archival repos being extremely slow and painful to work with directly. I'm not sure what the timeline for this is at the moment, but our team is actively working on getting this pipeline automated so exports are up-to-date. |
Beta Was this translation helpful? Give feedback.
-
|
could i for example write the csv data via udp to a consumer? for us it's more about how to consume the data properly into the transformation pipeline to make it read friendly/query'able i neither would need the postgres nor the csv file if i could plugin consumers for the data - i rather cache the stream into ram than having it persisted in a read unfriendly way |
Beta Was this translation helpful? Give feedback.
-
|
That's basically what Telegraf is doing. It's an ETL agent which can input/output/process data into a time-series dataset. If you need to "stream" CSV data, I'm pretty sure telegraf can do that if your consumer can keep up (telegraf's operational pattern is to push until pressure and then drop data which might be a poor fit for your case) but writing something to parse CSVs on newlines and stream to a connection shouldn't be too much of a lift. I think a plugin for Lily would end up being the solution you're looking for. The plugin would consume the Lily models directly or transforms/streams your data to another consumer. Then it just comes down to how you prefer to handle the Lily data within the plugin. (My best idea would be to pass in the raw models which match the DB schema as they are exported from the chain.) |
Beta Was this translation helpful? Give feedback.
-
We've been talking about the idea of leveraging golang plugins to allow lily to support custom scraping logic. Lily has developed into a robust distributed worker pipeline for scraping data/metrics from the Filecoin network. Lily is a full Lotus node, the reference node implementation on the Filecoin network, and continually re-integrates new releases as they are made available.
The targets are prioritized for inclusion by the Sentinel team (albeit with generous room for community input). But we'd like to remove ourselves as gatekeepers and enable others to build custom scrapers which can leverage Lily's existing worker pipeline and accurate view into Filecoin network's state and behavior.
Here are some potential use cases:
Such use cases serve a very narrow audience and wouldn't make sense to directly include in releases, but still have value and users should be able to easily collect this information without re-inventing the
wheelFilecoin network parsing machine.Would this be useful? How would you use it? How important is this to you? Other thoughts welcome, too! 🙏
Beta Was this translation helpful? Give feedback.
All reactions