Opensource social media intelligence and OSINT
Explore the docs »
View Demo
·
Report Bug
·
Request Feature
Table of Contents
There were 150 million new social media users between April 2022 and April 2023 - that's a 3.2% increase year-over-year from the current 4.8 billion social media users worldwide, representing 59.9% of the global population and 92.7% of all internet users. Businesses use social media to gain insights about various topics: user sentiment around products, strong products and weak products, events, all things that cater to their customer. For intelligence analysts and social science researchers, who are their customers? Policymakers, the everyday citizen, everyone in society. This project's user, its customer, is the intelligence and social sciences analysts and researchers. As technology continues up the curve of innovation, and society continues to increasingly use social media as the public square, researchers can use this available data for good, to draw insights, to slow down or stop harmful incidents, to help society, to develop plans based on public consensus, better inform policymakers of what their constituents need and want (and better plan solutions that increase the satisfaction of their customer). As it becomes harder for governments to understand and execute solutions that better serve their constituents, the idea of adaptive governance, a focus on decentralized decision-making structures, becomes unavoidable. Not only can insights gained from social media assist policymakers, it can assist adaptive governance entities and groups to better serve their populations. This product is for the intelligence analyst, the social scientist, the data scientist and those interested in improving the quality of our human existence through deep public analysis and data-driven solutions.
Along with other features, OpenSquare provides Digit Footprint targeting tools, some using well-known OSINT methods such as Backlinks, NSLookup, and Whois. Having a general suite of tools in a single place can increase user productivity. Easily navigate between dashboard workspaces and use output from one tool as input for another.
Opensquare utilizes Whisper: a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multitasking model that can perform multilingual speech recognition, speech translation, and language identification.
Using Opensquare's available API's, you can query and transcribe YouTube videos. Transcripts will report time and text properties. This API is used to build features on Opensquare , but will also be available to the public as an easy to use API.
opensquare/api/youtube/en/transcribe?videoId=l9AzO1FMgM8
produces:
[ { "time": "0.0", "text": "Java, a high-level multi-paradigm programming language famous for its ability to compile" }, { "time": "5.2", "text": "to platform independent bytecode." }, { "time": "7.44", "text": "It was designed by James Gosling in 1990 at Sun Microsystems." }, { "time": "11.700000000000001", "text": "One of its first demonstrations was the Star 7 PDA, which gave birth to the Java mascot" },... ]
To get a local copy up and running follow these simple example steps.
-
Linux
-
Java 17
java --version
-
Maven 3.9 or above
mvn --version
-
Clone the repo
git git clone https://opensentop@bitbucket.org/intelligence-opensent/opensentop.git
-
Install dependencies (including NPM) default profile
mvn clean install
-
Run webpack in development mode
npm run watch
-
There are some configuration files you'll need - feel free to ping me for those.
This project uses Eirik Sletteberg's Frontend-Maven-Plugin which allows our team to use a single plugin for both frontend and backend builds in a single repo.
This plugin is capable of various configurations, but the configuration used in this project is minimum only using Webpack and few configurations to install Node and NPM. The meat of this usage is from the
creation of the project's bundle which is integrated using a <script> in root of React application (typical React fashion) exposed in the index.html file in the Springboot resources folder.
<body>
<div id='root'>
</div>
<script src="built/bundle.js"></script>
</body>
Webpack will create a build bundle that contains the source for the React application entry in app.js under the js package of this project.
entry: path.resolve(__dirname, "/src/main/js/app.js"),
devtool: 'inline-source-map',
cache: true,
mode: 'development',
output: {
path: __dirname,
filename: 'src/main/resources/static/built/bundle.js'
},
If you're running kafka you should review the Docs. First ensure zoo-keeper server is running before running the kafka server. Sometimes zookeeper configs
/config folder is not setup properly. If you need to, ensure clientPort=2181 is set in the zookeeper.properties and to ensure non-conflicting ports make sure admin.serverPort=8083 is set in the same file.
We also want to ensure that bootstrap.servers=9092 is configured in producer.properties: this is a list of brokers used for bootstrapping knowledge about the rest of the cluster format which is important for
this project's springboot configuration below:
@Bean
public ConsumerFactory<String, OpenSentTaskStatus> consumerFactory() {
Map<String, Object> configurationProperties = new HashMap<>();
configurationProperties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
configurationProperties.put(ConsumerConfig.GROUP_ID_CONFIG, "group_id");
configurationProperties.put(JsonDeserializer.TRUSTED_PACKAGES, "*");
configurationProperties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
configurationProperties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
configurationProperties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
return new DefaultKafkaConsumerFactory<>(configurationProperties);
}
@Bean
public ProducerFactory<String, OpenSentTaskStatus> producerFactory() {
Map<String, Object> props = new HashMap<>();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
return new DefaultKafkaProducerFactory<>(props);
}
- YouTube Service & View
- X (formerly Twitter) Service & View
- Vkontakte Service & View
See the open issues for a full list of proposed features (and known issues).
Wali Morris - @LinkedIn - walimmorris@gmail.com
Project Link: GitHub