October 2019 – Data Science Blog (English only)

Why Retailers Are Making the Push for Stronger Data Science and AI

October 30, 2019/0 Comments/in Insights, Use Cases /by Kaylae Matthews

Retail relies on what the customer wants and needs at that moment, no matter the size of the company. Making judgments without consumer input would probably work for a little while but will fall flat as soon as the business model becomes outdated. In today’s technology-run world, things can become obsolete in a matter of days or even hours.

Retailers are the businesses most in need of capitalizing on what the customer wants in real-time. They have started to use data science and information from the Internet of Things (IoT) to not only stay in business, but also get ahead of other brands.

Artificial intelligence (AI) adds a new layer by using modern technology. The details of why retailers want to use these new practices are a bit more specific, though.

Data Targets Audiences

By using current customer data compared to information from the IoT, retailers can learn more about their audience and find better means of targeting them. Demographics like age, location and many other factors could affect advertising and even shopping, not to mention holidays throughout the year an audience celebrates.

Websites also need to be customized to suit the target audience. Those that are mobile-friendly and focused on what shoppers want can increase revenue, but the wrong approach can drive away new and existing customers. AI can help companies understand that data and present it back to the customer seamlessly, providing different options for various audiences.

Customer Base Expansion

Customer success should mean business success, as well. Growing a client base is something data science can assist with. However, helping customers grow is another type of service few companies provide but all people appreciate. A business can expand by offering new products and services that are relevant to their audience through the use of data.

Once a company learns what current customers want and begin to fit their needs, it can expand to more audiences. With data science, a business can ensure it does so slowly to give more of what current customers want while also finding new ones. The data can tell what sort of interests they all share so companies can capitalize on the venture.

AI Helps Customer Service

AI helps out customer service on both ends. Employees don’t have to focus on common problems that could easily be resolved, and clients often walk away happier than if they were to speak to a real person. This doesn’t work for every problem, especially ones that are specific in nature, but they can assist with more common issues. This is where chatbots enter the stage.

An AI-supported chatbot can give immediate support, provide suggestions, answer direct questions and offer almost any other form of help needed. Customers get personalized attention, and businesses can work faster toward customer loyalty.

Again, speaking to a real person when they have problems is a big plus for customers, but not for issues they know could be resolved in the time it takes to wait on the line for a representative.

Supply and Demand

Price optimization has taken on a bigger role than it has in the past. Mostly, data science is looking at supply and demand in real-time rather than having price fluctuations occur months after the business loses money. Having the right price can also help create more promotions for products and services, rewarding loyal customers for their shopping.

The data has to be gained from multiple channels by using price optimization tools, which focus on using data correctly in a company’s favor. The information doesn’t just look at supply and demand, but also examines locations, times, customer attitudes, competitor pricing and many other factors. All these pieces of information can be delivered in real-time so prices can be changed accordingly.

Taking the Competition

The thing about data science is that businesses are already utilizing it to their full potential and getting more customers than ever. The only way to get ahead of the competition is to at least start using the tools they’ve had at their disposal for years.

Target was one such company that took up the data helm. During 2012 and 2013, it saw a pretty sizeable dip in sales, but its online sales went up by almost 30% during the same time.

Data and Retail

When running a retail business, especially one that’s branching off into a franchise, using data is imperative. Data science and AI have become extremely important to companies both big and small.

Applying it correctly can help enterprises of any size and in every industry take things to the next level.

Even if a company is just starting out, sticking the first landing with a target audience is a fantastic way to begin the adventure and find success.

How to Make Better Decisions

October 22, 2019/0 Comments/in Uncategorized /by Signavio

Humans make decisions all the time. Some of these decisions are minor, like what to wear or what to eat. Some may seem minor, but actually have the potential to make a huge difference in an individual’s life; deciding if it’s safe to cross the road, for example. Of course, as the relative power of a decision-maker grows, the larger the impact, with many decisions affecting whole communities, or even the world.

Read this article in German:

Treffen Sie bessere Entscheidungen

In the same way, businesses depend on decisions. In fact, any business can be considered as the sum total of all sorts of decisions, large and small, from what new markets to enter into, to the next big advertising campaign, or what color to paint the walls in the new office. In an ideal world, each individual decision within an organization would be just one part of a consistent, coherent strategy driving the entire business.

Unfortunately, for many businesses, this consistency can be quite elusive. It can be difficult just to keep track of what was decided in yesterday’s meeting, let alone weeks, months, or years ago. One way to overcome this challenge is to identify, categorize, and standardize decision-making within your organization.

Strategic, tactical, and operational decisions

In broad terms, there are three ‘levels’ of decisions within a business. Strategic decisions are big-picture, concerning the company as a whole; things like mergers and acquisitions, or eliminating an underperforming line of business. Tactical decisions are those made on specific issues, like where and how to conduct a marketing campaign.

Finally, there are operational decisions, the kind every person in every company makes every day about the way they carry out their work. Examples of operational decisions include how many loyalty points to award a customer, which vendor to purchase materials and services from, how much credit to extend a customer, and many others. Millions of these decisions happen every day.

The cumulative effect of these operational decisions has huge impacts on business performance. Not necessarily on the broader issues facing a company, the way strategic or tactical decisions do, but on how smoothly and effectively things actually get done within the organization.

Risks of poor decision-making

At the operational level, even seemingly small decisions, if they are replicated widely, can have significant repercussions across a business. In many cases, this will mean:

Reduced operational compliance: employees and systems won’t know what management expects, or what the correct procedure is. In time this may lead to a general failure to comply with directives.
Less agility: unmanaged or unstructured decisions are difficult to change quickly in response to new internal or external circumstances.
Reduced accuracy: without a clear decision-making framework, inaccurate and imprecise targeting of process and practices may become more widespread.
Lack of transparency: employees and management may not be able to see and understand the factors that need to be taken into account for effective decision-making.
Increased regulatory non-compliance: many decisions affect tax, finance and environmental reporting, where the wrong choice leads to potentially breaking laws and regulations, and the resulting fines and legal costs.

These risks can manifest when decisions are not separated from the daily stream of business requirements. If the “right” decision can only be determined by searching through artifacts like use cases, stories, and processes, or relevant rules and data are spread across different parts of the business, then it is no surprise if that decision is difficult to reach.

How to make better decisions

The right decision at the right time is critical to business success, yet few businesses manage their decisions as separate entities. While most companies use KPI’s (or an equivalent) to measure the impact of their decisions, it is much less common for a business to create an inventory of the decisions themselves.

To overcome this, organizations should consider their important decisions as assets to be managed, just like any other business asset. The most effective way to do this is to make use of Business Decision Management, or BDM, a discipline used to identify, catalogue, and model decisions, particularly the operational decisions discussed above. BDM can also quantify their impact on performance and creates metrics and key indicators for the decisions.

With an effective BDM approach, businesses can then create models of their decisions, and more importantly the way they make decisions, using with Decision Model and Notation (DMN). DMN provides a clear, easy-to-follow notation system that describes business decisions, including the rules and data that drive the decision.

Better decisions with Signavio

The Signavio Business Transformation Suite offers a range of tools not only to support the DMN standard, but also to build a comprehensive environment for collaborating on the discovery, management and improvement of your decisions.

In particular, Signavio Process Manager gives you the capability to standardize, replicate, and re-use decisions across multiple business areas, as well as connecting those decisions to the business processes they drive. Signavio Process Manager empowers everyone in your organization to make the best decision for their work, no matter how complex.

Extracting the decision from the clutches of uncertain management and technology will reap many benefits, including improved performance and reduced risk. If you’d like to discover these benefits for yourself, why not sign up for a free 30 day trial with Signavio, today. Would you like to know more? Read our white paper on DMN.

c/o data science – in care of data science

October 19, 2019/in Events, Recommendations /by events

Are you looking for a platform where you can personally exchange ideas with other data scientists and data geeks and share and discuss challenges?

c/o data science is designed for YOU – be part of the premiere on November 12, 2019 at the Basecamp in Bonn! You can expect deep dive talks, hack sessions and a bar camp as well as live demos, code to go and lots of time for networking – create your own program!

Join us and share your passion with peers!

And who’s running the camp?

SIGS DATACOM is an international and vendor-independent company for further education in information technology. SIGS DATACOM is a leading provider of further education in the fields of software architecture and engineering, data and insights as well as artificial intelligence. SIGS DATACOM offers high-quality specialist information to software architects, IT project managers/managers, experienced programmers, developers and business intelligence/analytics professionals, project managers and consultants as well as AI professionals and data scientists. The c/o data science is a consistent addition to our previous offering in the area of continuing education and specialist information.

All information: https://www.co-datascience.de/

Datum 12.11.2019

Organizer: Sigz Datacom GmbH

Venue:

Basecamp Bonn

In der Raste 1

Bonn, 53129 Germany

https://www.co-datascience.de/

Cost: zwsichen 30-300€

Glorious career paths of a Big Data Professional

October 18, 2019/0 Comments/in Big Data, Business Analytics, Data Engineering, Data Security /by Digvijay Upadhyay

Are you wondering about the career profiles you may get to fill if you get into Big Data industry? If yes, then Bingo! This is the post that will inform you just about that. Big data is just an umbrella term. There are a lot of profiles and career paths that are covered under this umbrella term. Let us have a look at some of these profiles.

Data Visualisation Specialist

The process of visualizing data is turning out to be critical in guaranteeing information-driven representatives get the upfront investment required to actualize goal-oriented and significant Big Data extends in their organization. Making your data to tell a story and the craft of envisioning information convincingly has turned into a significant piece of the Big Data world and progressively associations need to have these capacities in-house. Besides, as a rule, these experts are relied upon to realize how to picture in different instruments, for example, Spotfire, D3, Carto, and Tableau – among numerous others. Information Visualization Specialists should be versatile and inquisitive to guarantee they stay aware of most recent patterns and answers for a recount to their information stories in the most intriguing manner conceivable with regards to the board room.

Big Data Architect

This is the place the Hadoop specialists come in. Ordinarily, a Big Data planner tends to explicit information issues and necessities, having the option to portray the structure and conduct of a Big Data arrangement utilizing the innovation wherein they practice – which is, as a rule, mostly Hadoop.

These representatives go about as a significant connection between the association (and its specific needs) and Data Scientists and Engineers. Any organization that needs to assemble a Big Data condition will require a Big Data modeler who can serenely deal with the total lifecycle of a Hadoop arrangement – including necessity investigation, stage determination, specialized engineering structure, application plan, and advancement, testing the much-dreaded task of deploying lastly.

Systems Architect

This Big data professional is in charge of how your enormous information frameworks are architected and interconnected. Their essential incentive to your group lies in their capacity to use their product building foundation and involvement with huge scale circulated handling frameworks to deal with your innovation decisions and execution forms. You’ll need this individual to construct an information design that lines up with the business, alongside abnormal state anticipating the improvement. The person in question will consider different limitations, adherence to gauges, and varying needs over the business.

Here are some responsibilities that they play:

- Determine auxiliary prerequisites of databases by investigating customer tasks, applications, and programming; audit targets with customers and assess current frameworks.
- Develop database arrangements by planning proposed framework; characterize physical database structure and utilitarian abilities, security, back-up and recuperation particulars.
- Install database frameworks by creating flowcharts; apply ideal access methods, arrange establishment activities, and record activities.
- Maintain database execution by distinguishing and settling generation and application advancement issues, figuring ideal qualities for parameters; assessing, incorporating, and putting in new discharges, finishing support and responding to client questions.
- Provide database support by coding utilities, reacting to client questions, and settling issues.

Artificial Intelligence Developer

The certain promotion around Artificial Intelligence is additionally set to quicken the number of jobs publicized for masters who truly see how to apply AI, Machine Learning, and Deep Learning strategies in the business world. Selection representatives will request designers with broad learning of a wide exhibit of programming dialects which loan well to AI improvement, for example, Lisp, Prolog, C/C++, Java, and Python.

All said and done; many people estimate that this popular demand for AI specialists could cause a something like what we call a “Brain Drain” organizations poaching talented individuals away from the universe of the scholarly world. A month ago in the Financial Times, profound learning pioneer and specialist Yoshua Bengio, of the University of Montreal expressed: “The industry has been selecting a ton of ability — so now there’s a lack in the scholarly world, which is fine for those organizations. However, it’s not extraordinary for the scholarly world.” It ; howeverusiasm to perceive how this contention among the scholarly world and business is rotated in the following couple of years.

Data Scientist

The move of Big Data from tech publicity to business reality may have quickened, yet the move away from enrolling top Data Scientists isn’t set to change in 2020. An ongoing Deloitte report featured that the universe of business will require three million Data Scientists by 2021, so if their expectations are right, there’s a major ability hole in the market. This multidisciplinary profile requires specialized logical aptitudes, specialized software engineering abilities just as solid gentler abilities, for example, correspondence, business keenness, and scholarly interest.

Data Engineer

Clean and quality data is crucial in the accomplishment of Big Data ventures. Consequently, we hope to see a lot of opening in 2020 for Data Engineers who have a predictable and awesome way to deal with information transformation and treatment. Organizations will search for these special data masters to have broad involvement in controlling data with SQL, T-SQL, R, Hadoop, Hive, Python and Spark. Much like Data Scientists. They are likewise expected to be innovative with regards to contrasting information with clashing information types with have the option to determine issues. They additionally frequently need to make arrangements which enable organizations to catch existing information in increasingly usable information groups – just as performing information demonstrations and their modeling.

IT/Operations Manager Job Description

In Big data industry, the IT/Operations Manager is a profitable expansion to your group and will essentially be in charge of sending, overseeing, and checking your enormous information frameworks. You’ll depend on this colleague to plan and execute new hardware and administrations. The person in question will work with business partners to comprehend the best innovation ventures to address their procedures and concerns—interpreting business necessities to innovation plans. They’ll likewise work with venture chiefs to actualize innovation and be in charge of effective progress and general activities.

Here are some responsibilities that they play:

Manage and be proactive in announcing, settling and raising issues where required
Lead and co-ordinate issue the executive’s exercises, notwithstanding ceaseless procedure improvement activities
Proactively deal with our IT framework
Supervise and oversee IT staffing, including enrollment, supervision, planning, advancement, and assessment
Verify existing business apparatuses and procedures remain ideally practical and worth included
Benchmark, dissect, report on and make suggestions for the improvement and development of the IT framework and IT frameworks
Advance and keep up a corporate SLA structure

Conclusion

These are some of the best career paths that big data professionals can play after entering the industry. Honesty and hard work can always take you to the zenith of any field that you choose to be in. Also, keep upgrading your skills by taking newer certifications and technologies. Good Luck

Visual Question Answering with Keras – Part 2: Making Computers Intelligent to answer from images

October 2, 2019/0 Comments/in Artificial Intelligence, Data Science, Data Science Hack, Insights, Main Category, Predictive Analytics, Uncategorized /by Akshay Chavan

Making Computers Intelligent to answer from images

This is my second blog on Visual Question Answering, in the last blog, I have introduced to VQA, available datasets and some of the real-life applications of VQA. If you have not gone through then I would highly recommend you to go through it. Click here for more details about it.

In this blog post, I will walk through the implementation of VQA in Keras.

You can download the dataset from here: https://visualqa.org/index.html. All my experiments were performed with VQA v2 and I have used a very tiny subset of entire dataset i.e all samples for training and testing from the validation set.

Table of contents:

Preprocessing Data
Process overview for VQA
Data Preprocessing – Images
Data Preprocessing through the spaCy library- Questions
Model Architecture
Defining model parameters
Evaluating the model
Final Thought
References

NOTE: The purpose of this blog is not to get the state-of-art performance on VQA. But the idea is to get familiar with the concept. All my experiments were performed with the validation set only.

Full code on my Github here.

1. Preprocessing Data:

If you have downloaded the dataset then the question and answers (called as annotations) are in JSON format. I have provided the code to extract the questions, annotations and other useful information in my Github repository. All extracted information is stored in .txt file format. After executing code the preprocessing directory will have the following structure.

All text files will be used for training.

2. Process overview for VQA:

As we have discussed in previous post visual question answering is broken down into 2 broad-spectrum i.e. vision and text. I will represent the Neural Network approach to this problem using the Convolutional Neural Network (for image data) and Recurrent Neural Network(for text data).

If you are not familiar with RNN (more precisely LSTM) then I would highly recommend you to go through Colah’s blog and Andrej Karpathy blog. The concepts discussed in this blogs are extensively used in my post.

The main idea is to get features for images from CNN and features for the text from RNN and finally combine them to generate the answer by passing them through some fully connected layers. The below figure shows the same idea.

Image Source: https://arxiv.org/pdf/1505.00468.pdf

I have used VGG-16 to extract the features from the image and LSTM layers to extract the features from questions and combining them to get the answer.

3. Data Preprocessing – Images:

Images are nothing but one of the input to our model. But as you already may know that before feeding images to the model we need to convert into the fixed-size vector.

So we need to convert every image into a fixed-size vector then it can be fed to the neural network. For this, we will use the VGG-16 pretrained model. VGG-16 model architecture is trained on millions on the Imagenet dataset to classify the image into one of 1000 classes. Here our task is not to classify the image but to get the bottleneck features from the second last layer.

Hence after removing the softmax layer, we get a 4096-dimensional vector representation (bottleneck features) for each image.

Image Source: https://www.cs.toronto.edu/~frossard/post/vgg16/

For the VQA dataset, the images are from the COCO dataset and each image has unique id associated with it. All these images are passed through the VGG-16 architecture and their vector representation is stored in the “.mat” file along with id. So in actual, we need not have to implement VGG-16 architecture instead we just do look up into file with the id of the image at hand and we will get a 4096-dimensional vector representation for the image.

4. Data Preprocessing through the spaCy library- Questions:

spaCy is a free, open-source library for advanced Natural Language Processing (NLP) in Python. As we have converted images into a fixed 4096-dimensional vector we also need to convert questions into a fixed-size vector representation. For installing spaCy click here.

You might know that for training word embeddings in Keras we have a layer called an Embedding layer which takes a word and embeds it into a higher dimensional vector representation. But by using the spaCy library we do not have to train the get the vector representation in higher dimensions.

This model is actually trained on billions of tokens of the large corpus. So we just need to call the vector method of spaCy class and will get vector representation for word.

After fitting, the vector method on tokens of each question will get the 300-dimensional fixed representation for each word.

5. Model Architecture:

In our problem the input consists of two parts i.e an image vector, and a question, we cannot use the Sequential API of the Keras library. For this reason, we use the Functional API which allows us to create multiple models and finally merge models.

The below picture shows the high-level architecture idea of submodules of neural network.

After concatenating the 2 different models the summary will look like the following.

The below plot helps us to visualize neural network architecture and to understand the two types of input:

6. Defining model parameters:

The hyperparameters that we are going to use for our model is defined as follows:

If you know what this parameter means then you can play around it and can get better results.

Time Taken: I used the GPU on https://colab.research.google.com and hence it took me approximately 2 hours to train the model for 5 epochs. However, if you train it on a PC without GPU, it could take more time depending on the configuration of your machine.

7. Evaluating the model:

Since I have used the very small dataset for performing these experiments I am not able to get very good accuracy. The below code will calculate the accuracy of the model.

Since I have trained a model multiple times with different parameters you will not get the same accuracy as me. If you want you can directly download mode.h5 file from my google drive.

8. Final Thoughts:

One of the interesting thing about VQA is that it a completely new field. So there is absolutely no end to what you can do to solve this problem. Below are some tips while replicating the code.

Start with a very small subset of data: When you start implementing I suggest you start with a very small amount of data. Because once you are ready with the whole setup then you can scale it any time.
Understand the code: Understanding code line by line is very much helpful to match your theoretical knowledge. So for that, I suggest you can take very few samples(maybe 20 or less) and run a small chunk (2 to 3 lines) of code to get the functionality of each part.
Be patient: One of the mistakes that I did while starting with this project was to do everything at one go. If you get some error while replicating code spend 4 to 5 days harder on that. Even after that if you won’t able to solve, I would suggest you resume after a break of 1 or 2 days.

VQA is the intersection of NLP and CV and hopefully, this project will give you a better understanding (more precisely practically) with most of the deep learning concepts.

If you want to improve the performance of the model below are few tips you can try: