Persisting Trained Machine Learning Models Using MsgPack

Introduction

At OpenSooq, we take AI seriously. Tensroflow and Scikit-Learn are two examples of popular packages used for machine learning, that are used in OpenSooq for different aspects of our e-commerce platform and its supporting services and moderation tools. In Supervised Machine Learning there are two phases training (fitting the model) and prediction. Training is done offline, then models get evaluated and then persisted to be deployed on production. Tensroflow models can be persisted as frozen protocol buffer files, on

Continue Reading

On-the-fly ad-hoc docker compose development stack

The problem

In this blog post we are going to discuss an application stack that uses MongoDB, Redis and NodeJS. We want a single docker compose command that launches the entire stack, still we want to the developer be able to use his/her favorite IDE to edit the source code without having to rebuild docker images. We call this on-the-fly ad-hoc stack because it does not involve building any docker images or hosting a docker registry, the reason for this

Continue Reading

Availability-zone-aware DNS Service-Discovery with DNSMasq and Ansible

In opensooq we heavily implement SOA (Service-oriented architecture) and we have a lot of Microservices. And since we have a very scalable dynamic environment that keeps changing, we need some sort of service discovery.

DNS Service Discovery allow us to access our microservices and supportive services by name. It’s not an alternative to load-balancing but a complement to that. As a rule of thumb we place two load-balancers in each availability, and we have DNS round-robin load-balancing for the load

Continue Reading

Manage your Docker image layers with Ansible

If you are managing infrastructure at large scale as we do in OpenSooq you had better have good level of automation and orchestration, your SysAdmins and operators had better handle their infrastructure as code. In OpenSooq, our preferred tool is Ansible.

Ansible is exceptionally great to put your servers into the desired state, which make it ideal for deployment orchestration and configuration management. For containerized environment, typically people use Dockerfiles to describe their setup inside the container, so they

Continue Reading

Replicating your SQL into NoSQL and full text search engine (Solr/ElasticSearch)

Opensooq uses so many open source technologies, because each piece has it’s own use case, for example we use Apache Solr to power the content moderation dashboard.

When your main data source is a strict classical ACID SQL, and you want to take that data to some other backend like NoSQL’s, ElasticSearch, or Solr as in our case, for sure you will get out of sync data. Keeping them in-sync becomes harder and

Continue Reading

Binding two SystemD-services to manage Loadbalancer IPVS

We have demonstrated how to make scalable resilient UDP service behind  IPVS load-balancer.

In this article we will see how we can bind two systemd services (one is the process on a given port and the other is registering it to IPVS)

Requires, Before and After

When you write systemd unit files for two dependent systemd services, let’s say A then B, usually we specify both “Requires and “After” for example, in B.service you specify “Requires=A.service” and “After=A.service” which

Continue Reading

Scalable Highly-available Resilient UDP Service using IPVS and smart client

The problem

We wanted a service that collects datapoints pass them to some backend to be queried later. We don’t want to bock senders of data or reduce their speed waiting their data to be submitted.

We have made a version of this service that is exposed via UDP, the service respond with “1” to acknowledge client that datapoint is received successfully because UDP unlike TCP have no acknowledgment.

Although UDP is much lightweight compared to TCP-based (including HTTP), it’s

Continue Reading

Site Footer

@ OpenSooq 2019