Emilia Cheladze

Article series: 5 Clean Coding Tips – 5.Put yourself in somebody else’s shoes

/0 Comments/in , , /by
This is the fifth of the article series “5 tips for clean coding” to follow as soon as you’ve made the first steps into your coding career, in this article series. Read the introduction here, to find out why it is important to write clean code if you missed it.

It might be a bit repetitive to bring up how important the readability of the code is, let’s do it anyway. In the majority of the cases you are writing for others, therefore you need to put yourself in their shoes to be able to assess how good the readability of your code is. For you, it all might be obvious because you wrote it. But it doesn’t have to be easy to read for someone else. If you have a colleague or a friend that has a bit of time for you and is willing to give you feedback, that is great. If, however, you don’t have such a person, having a few imaginary friends might be helpful in this case. It might sound crazy, but don’t close this page just yet. Having a set of imaginary personas at your disposal, to review your work with their eyes, can help you a lot. Imagine that your code met one of those guys. What would they say about it? If you work in a team or collaborate with people, you probably don’t have to imagine them. You’ve met them.

The_PEP8_guy – He has years of experience. He is used to seeing the code in a very particular way. He quotes the style guide during lunch. His fingers make the perfect line splitting and indentation without even his thoughts reaching the conscious state. He knows that lowercase_with_underscore is for variables, UPPER_CASE_NAMES are for constants and the CapitalizedWords are for classes. He will be lost if you do it in any different way. His expectations will not meet what you wrote, and he will not understand anything, because he will be too distracted by the messed up visual. Depending on the character he might start either crying or shouting. Read the style guide and follow it. You might be able to please this guy at least a little bit with the automatic tools like pylint.

The_ grieving _widow – Imagine that something happens to you. Let’s say, that you get hit by a bus[i]. You leave behind sadness and the_ grieving_widow to manage your code, your legacy. Will the future generations be able to make use of it or were you the only one who can understand anything you wrote? That is a bit of an extreme situation, ok. Alternatively, imagine, that you go for a 5-week vacation to a silent retreat with a strict no-phone policy (or that is what you tell your colleagues). Will they be able to carry on if they cannot ask you anything about the code? Review your code and the documentation from the perspective of the poor grieving_widow.

The_not_your_domain_guy – He is from the outside of the world you are currently in and he just does not understand your jargon. He doesn’t have to know that in data science a feature, a predictor and an x probably mean the same thing. SNR might shout signal-to-noise ratio at you, it will only snort at him. You might use abbreviations that are obvious to you but not to everyone. If you think that the majority of people can understand, and it helps with the code readability keep the abbreviations but just in case, document/comment them. There might be abbreviations specific to your company and, someone from the outside, a new guy, a consultant will not get them. Put yourself in the shoes of that guy and maybe make your code a bit more democratic wherever possible.

The_foreigner– You might be working in an environment, where every single person speaks the same language you speak, and it happens not to be English. So, you and your colleagues name variables and write the comments in your language. However, unless you work in a team with rules a strict as Athletic Bilbao, there might be a foreigner joining your team in the future. It is hard to argue that English is the lingua franca in programming (and in the world), these days. So, it might be worth putting yourself in the_foreigner’s shoes, while writing your code, to avoid a huge amount of work in the future, that the translation and explanation will require. And even if you are working on your own, you might want to make your code public one day and want as many people as possible to read it.

The_hurry_up_guy – we all know this guy. Sometimes he doesn’t have a body or a face, but we can feel his presence. You might want to write a perfect solution, comment it in the best possible way and maybe add a bit of glitter on top but sometimes you just need to give in and do it his way. And that’s ok too.

References:

[i] https://en.wikipedia.org/wiki/Bus_factor

Emilia Cheladze

Article series: 5 Clean Coding Tips – 4. Stop commenting the obvious

/0 Comments/in , , /by
This is the fourth of the article series “5 tips for clean coding” to follow as soon as you’ve made the first steps into your coding career, in this article series. Read the introduction here, to find out why it is important to write clean code if you missed it.

Everyone will tell you that you need to comment your code. You do it for yourself, for others, it might help you to put down a structure of your code before you get down to coding properly. Writing a lot of comments might give you a false sense of confidence, that you are doing a good job. While in reality, you are commenting your code a lot with obvious, redundant statements that are not bringing any value. The role of a comment it to explain, not to describe. You need to realize that any piece of comment has to add information to the code you already have, not to double it.

Keep in mind, you are not narrating the code, adding ‘subtitles’ to python’s performance. The comments are there to clarify what is not explicit in the code itself. Adding a comment saying what the line of code does is completely redundant most of the time:

A good rule of thumb would be: if it starts to sound like an instastory, rethink it. ‘So, I am having my breakfast, with a chai latte and my friend, the cat is here as well’. No.

It is also a good thing to learn to always update necessary comments before you modify the code. It is incredibly easy to modify a line of code, move on and forget the comment. There are people who claim that there are very few crimes in the world worse than comments that contradict the code itself.

Of course, there are situations, where you might be preparing a tutorial for others and you want to narrate what the code is doing. Then writing that load function will load the data is good. It does not have to be obvious for the listener. When teaching, repetitions, and overly explicit explanations are more than welcome. Always have in mind who your reader will be.

Hoang Tu Nguyen

Ein Einblick in die Aktienmärkte unter Berücksichtigung von COVID-19

/0 Comments/in , , , , /by

Einleitung

Die COVID-19-Pandemie hat uns alle fest im Griff. Besonders die Wirtschaft leidet stark unter den erforderlichen Maßnahmen, die weltweit angewendet werden. Wir wollen daher die Gelegenheit nutzen einen Blick auf die Aktienkurse zu wagen und analysieren, inwieweit der Virus einen Einfluss auf das Wachstum des Marktes hat.

Rahmenbedingungen

Zuallererst werden wir uns auf die Industrie-, Schwellenländer und Grenzmärkte konzentrieren. Dafür nutzen wir die MSCI Global Investable Market Indizes (kurz GIMI), welche die zuvor genannten Gruppen abbilden. Die MSCI Inc. ist ein US-amerikanischer Finanzdienstleister und vor allem für ihre Aktienindizes bekannt.

Aktienindizes sind Kennzahlen der Entwicklung bzw. Änderung einer Auswahl von Aktienkursen und können repräsentativ für ganze Märkte, spezifische Branchen oder Länder stehen. Der DAX ist zum Beispiel ein Index, welcher die Entwicklung der größten 30 deutschen Unternehmen zusammenfasst.

Leider sind die Daten von MSCI nicht ohne weiteres zugänglich, weshalb wir unsere Analysen mit ETFs (engl.: “Exchange Traded Fund”) durchführen werden. ETFs sind wiederum an Börsen gehandelte Fonds, die von Fondgesellschaften/-verwaltern oder Banken verwaltet werden.

Für unsere erste Analyse sollen folgende ETFs genutzt werden, welche die folgenden Indizes führen:

Index Beschreibung ETF
MSCI World über 1600 Aktienwerte aus 24 Industrieländern iShares MSCI World ETF
MSCI Emerging Markets ca. 1400 Aktienwerte aus 27 Schwellenländern iShares MSCI Emerging Markets ETF
MSCI Frontier Markets Aktienwerte aus ca. 29 Frontier-Ländern iShares MSCI Frontier 100 ETF

Tab.1: MSCI Global Investable Market Indizes mit deren repräsentativen ETFs

Datenquellen

Zur Extraktion der ETF-Börsenkurse nehmen wir die yahoo finance API zur Hilfe. Mit den richtigen Symbolen können wir die historischen Daten unserer ETF-Auswahl ausgeben lassen. Wie unter diesem Link für den iShares MSCI World ETF zu sehen ist, gibt es mehrere Werte in den historischen Daten. Für unsere Analyse nutzen wir den Wert, nachdem die Börse geschlossen hat.

Da die ETFs in ihren Kurswerten Unterschiede haben und uns nur die relative Entwicklung interessiert, werden wir relative Werte für die Analyse nutzen. Der Startzeitpunkt soll mit dem 06.01.2020 festgelegt werden.

Die Daten über bestätigte Infektionen mit COVID-19 entnehmen wir aus der Hochrechnung der Johns Hopkins Universität.

Correlation between confirmed cases and growth of MSCI GIMI
Abb.1: Interaktives Diagramm: Wachstum der Aktienmärkte getrennt in Industrie-, Schwellen-, Frontier-Länder und deren bestätigten COVID-19 Fälle über die Zeit. Die bestätigten Fälle der jeweiligen Märkte basieren auf der Aufsummierung der Länder, welche auch in den Märkten aufzufinden sind und daher kann es zu Unterschieden bei den offiziellen Zahlen kommen.

Interpretation des Diagramms

Auf den ersten Blick sieht man deutlich, dass mit steigenden COVID-19 Fällen die Aktienkurse bis zu -31% einbrechen. (Anfangszeitpunkt: 06.01.2020 Endzeitpunkt: 09.04.2020)

Betrachten wir den Anfang des Diagramms so sehen wir einen Einbruch der Emerging Markets, welche eine Gewichtung von 39.69 % (Stand 09.04.20) chinesische Aktien haben. Am 17.01.20 verzeichnen die Emerging Marktes noch ein Plus von 3.15 % gegenüber unserem Startzeitpunkt, wohingegen wir am 01.02.2020 ein Defizit von -6.05 % gegenüber dem Startzeitpunkt haben, was ein Einbruch von -9.20 % zum 17.01.2020 entspricht. Da der Ursprung des COVID-19 Virus auch in China war, könnte man diesen Punkt als Grund des Einbruches interpretieren. Die Industrie- und  Frontier-Länder bleiben hingegen recht stabil und auch deren bestätigten Fälle sind noch sehr niedrig.

Die Industrieländer erreichen ihren Höchststand am 19.02.20 mit einem Plus von 2.80%. Danach brachen alle drei Märkte deutlich ein. Auch in diesem Zeitraum gab es die ersten Todesopfer in Europa und in den USA. Der derzeitige Tiefpunkt, welcher am 23.03.20 zu registrieren ist, beläuft sich für die Industrieländer -32.10 %, Schwellenländer 31.7 % und Frontier-Länder auf -34.88 %.

Interessanterweise steigen die Marktwerte ab diesem Zeitpunkt wieder an. Gründe könnten die Nachrichten aus China sein, welche keine weiteren Neu-Infektionen verzeichnen, die FED dem Markt bis zu 1.5 Billionen Dollar zur Verfügung stellt und/oder die Ankündigung der Europäische Zentralbank Anleihen in Höhe von 750 MRD. Euro zu kaufen. Auch in Deutschland wurden große Hilfspakete angekündigt.

Um detaillierte Aussagen treffen zu können, müssen wir uns die Kurse auf granularer Ebene anschauen. Durch eine gezieltere Betrachtung auf Länderebene könnten Zusammenhänge näher beschrieben werden.

Wenn du dich für interaktive Analysen interessierst und tiefer in die Materie eintauchen möchtest: DATANOMIQ COVID-19 Dashboard

Hier haben wir ein Dashboard speziell für Analysen für die Aktienmärkte, welches stetig verbessert wird. Auch sollen Krypto-Währungen bald implementiert werden. Habt ihr Vorschläge und Verbesserungswünsche, dann lasst gerne ein Kommentar da!

Emilia Cheladze

Article series: 5 Clean Coding Tips – 3. Take Advantage of the Formatting Tools.

/0 Comments/in , , /by
This is the third of the article series “5 tips for clean coding” to follow as soon as you’ve made the first steps into your coding career, in this article series. Read the introduction here, to find out why it is important to write clean code if you missed it.

Unfortunately, no automatic formatting tool will correct the logic in your code, suggest meaningful names of your variables or comment the code for you. Yet. Gmail has lately started suggesting email titles based on email content. AI-powered variable naming can be next, who knows. Anyway, the visual level of the code is much easier to correct and there are tools that will do some of the code formatting on the visual level job for you. Some of them might be already existing in your IDE, you just need to look for them a bit, others need to be installed. One of the most popular formatting tools is pylint[i]. It is worth checking it out and learning to use it in an efficient way.

Beware that as convenient as it may seem to copy and paste your code into a quick online ‘beautifier’ it is not always a good idea. The online tools might store your code. If you are working on something that shouldn’t just freely float in the world wide web, stick to reliable tools like pylint, that will store the data within your working directory.

These tools can become very good friends of yours but also very annoying ones. They will not miss single whitespace and will not keep their mouth shut when your line length jumps from 79 to 80 characters. They will be shouting with an underscoring of some worrying color and/or exclamation marks. You will need to find your way to coexist and retain your sanity. It can be very distracting when you are in a working flow and warnings pop up all the time about formatting details that have nothing to do with what you are trying to solve. Sometimes, it might be better to turn those warnings off while you are in your most concentrated/creative phase of writing and turn them back on while the dust of your genius settles down a little bit. Usually the offer a lot of flexibility, regarding which warnings you want to be ignored and other features. The good thing is, they also teach you what are mistakes that you are making and after some time you will just stop making them in the first place.

References:

[i] https://www.pylint.org/

Hoang Tu Nguyen

Einführung in die Welt der Autoencoder

/0 Comments/in , , , , , , , , , /by

An wen ist der Artikel gerichtet?

In diesem Artikel wollen wir uns näher mit dem neuronalen Netz namens Autoencoder beschäftigen und wollen einen Einblick in die Grundprinzipien bekommen, die wir dann mit einem vereinfachten Programmierbeispiel festigen. Kenntnisse in Python, Tensorflow und neuronalen Netzen sind dabei sehr hilfreich.

Funktionsweise des Autoencoders

Ein Autoencoder ist ein neuronales Netz, welches versucht die Eingangsinformationen zu komprimieren und mit den reduzierten Informationen im Ausgang wieder korrekt nachzubilden.

Die Komprimierung und die Rekonstruktion der Eingangsinformationen laufen im Autoencoder nacheinander ab, weshalb wir das neuronale Netz auch in zwei Abschnitten betrachten können.

 

 

 

Der Encoder

Der Encoder oder auch Kodierer hat die Aufgabe, die Dimensionen der Eingangsinformationen zu reduzieren, man spricht auch von Dimensionsreduktion. Durch diese Reduktion werden die Informationen komprimiert und es werden nur die wichtigsten bzw. der Durchschnitt der Informationen weitergeleitet. Diese Methode hat wie viele andere Arten der Komprimierung auch einen Verlust.

In einem neuronalen Netz wird dies durch versteckte Schichten realisiert. Durch die Reduzierung von Knotenpunkten in den kommenden versteckten Schichten werden die Kodierung bewerkstelligt.

Der Decoder

Nachdem das Eingangssignal kodiert ist, kommt der Decoder bzw. Dekodierer zum Einsatz. Er hat die Aufgabe mit den komprimierten Informationen die ursprünglichen Daten zu rekonstruieren. Durch Fehlerrückführung werden die Gewichte des Netzes angepasst.

Ein bisschen Mathematik

Das Hauptziel des Autoencoders ist, dass das Ausgangssignal dem Eingangssignal gleicht, was bedeutet, dass wir eine Loss Funktion haben, die L(x , y) entspricht.

L(x, \hat{x})

Unser Eingang soll mit x gekennzeichnet werden. Unsere versteckte Schicht soll h sein. Damit hat unser Encoder folgenden Zusammenhang h = f(x).

Die Rekonstruktion im Decoder kann mit r = g(h) beschrieben werden. Bei unserem einfachen Autoencoder handelt es sich um ein Feed-Forward Netz ohne rückkoppelten Anteil und wird durch Backpropagation oder zu deutsch Fehlerrückführung optimiert.

Formelzeichen Bedeutung
\mathbf{x}, \hat{\mathbf{x}} Eingangs-, Ausgangssignal
\mathbf{W}, \hat{\mathbf{W}} Gewichte für En- und Decoder
\mathbf{B}, \hat{\mathbf{B}} Bias für En- und Decoder
\sigma, \hat{\sigma} Aktivierungsfunktion für En- und Decoder
L Verlustfunktion

Unsere versteckte Schicht soll mit \latex h gekennzeichnet werden. Damit besteht der Zusammenhang:

(1)   \begin{align*} \mathbf{h} &= f(\mathbf{x}) = \sigma(\mathbf{W}\mathbf{x} + \mathbf{B}) \\ \hat{\mathbf{x}} &= g(\mathbf{h}) = \hat{\sigma}(\hat{\mathbf{W}} \mathbf{h} + \hat{\mathbf{B}}) \\ \hat{\mathbf{x}} &= \hat{\sigma} \{ \hat{\mathbf{W}} \left[\sigma ( \mathbf{W}\mathbf{x} + \mathbf{B} )\right] + \hat{\mathbf{B}} \}\\ \end{align*}

Für eine Optimierung mit der mittleren quadratischen Abweichung (MSE) könnte die Verlustfunktion wie folgt aussehen:

(2)   \begin{align*} L(\mathbf{x}, \hat{\mathbf{x}}) &= \mathbf{MSE}(\mathbf{x}, \hat{\mathbf{x}}) = \| \mathbf{x} - \hat{\mathbf{x}} \| ^2 &= \| \mathbf{x} - \hat{\sigma} \{ \hat{\mathbf{W}} \left[\sigma ( \mathbf{W}\mathbf{x} + \mathbf{B} )\right] + \hat{\mathbf{B}} \} \| ^2 \end{align*}

 

Wir haben die Theorie und Mathematik eines Autoencoder in seiner Ursprungsform kennengelernt und wollen jetzt diese in einem (sehr) einfachen Beispiel anwenden, um zu schauen, ob der Autoencoder so funktioniert wie die Theorie es besagt.

Dazu nehmen wir einen One Hot (1 aus n) kodierten Datensatz, welcher die Zahlen von 0 bis 3 entspricht.

    \begin{align*} [1, 0, 0, 0] \ \widehat{=} \ 0 \\ [0, 1, 0, 0] \ \widehat{=} \ 1 \\ [0, 0, 1, 0] \ \widehat{=} \ 2 \\ [0, 0, 0, 1] \ \widehat{=} \ 3\\ \end{align*}

Diesen Datensatz könnte wie folgt kodiert werden:

    \begin{align*} [1, 0, 0, 0] \ \widehat{=} \ 0 \ \widehat{=} \ [0, 0] \\ [0, 1, 0, 0] \ \widehat{=} \ 1 \ \widehat{=} \ [0, 1] \\ [0, 0, 1, 0] \ \widehat{=} \ 2 \ \widehat{=} \ [1, 0] \\ [0, 0, 0, 1] \ \widehat{=} \ 3 \ \widehat{=} \ [1, 1] \\ \end{align*}

Damit hätten wir eine Dimensionsreduktion von vier auf zwei Merkmalen vorgenommen und genau diesen Vorgang wollen wir bei unserem Beispiel erreichen.

Programmierung eines einfachen Autoencoders

 

Typische Einsatzgebiete des Autoencoders sind neben der Dimensionsreduktion auch Bildaufarbeitung (z.B. Komprimierung, Entrauschen), Anomalie-Erkennung, Sequenz-to-Sequenz Analysen, etc.

Ausblick

Wir haben mit einem einfachen Beispiel die Funktionsweise des Autoencoders festigen können. Im nächsten Schritt wollen wir anhand realer Datensätze tiefer in gehen. Auch soll in kommenden Artikeln Variationen vom Autoencoder in verschiedenen Einsatzgebieten gezeigt werden.

Emilia Cheladze

Article series: 5 Clean Coding Tips – 2. Name Variables in a Meaningful Way

/0 Comments/in , , /by
This is the second of the article series “5 tips for clean coding” to follow as soon as you’ve made the first steps into your coding career, in this article series. Read the introduction here, to find out why it is important to write clean code if you missed it.

When it comes to naming variables, there are a few official rules in the PEP8 style guide. A variable must start with an underscore or a letter and can be followed by a number of underscores or letters or digits. They cannot be reserved words: True, False, or, not, lambda etc. The preferred naming style is lowercase or lowercase_with_underscore. This all refers to variable names on a visual level. However, for readability purposes, the semantic level is as important, or maybe even more so. If it was for python, the variables could be named like this:

It wouldn’t make the slightest difference. But again, the code is not only for the interpreter to be read. It is for humans. Other people might need to look at your code to understand what you did, to be able to continue the work that you have already started. In any case, they need to be able to decipher what hides behind the variable names, that you’ve given the objects in your code. They will need to remember what they meant as they reappear in the code. And it might not be easy for them.

Remembering names is not an easy thing to do in all life situations. Let’s consider the following situation. You go to a party, there is a bunch of new people that you meet for the first time. They all have names and you try very hard to remember them all. Imagine how much easier would it be if you could call the new girl who came with John as the_girl_who_came_with_John. How much easier would it be to gossip to your friends about her? ‘Camilla is on the 5th glass of wine tonight, isn’t she?!.’ ‘Who are you talking about???’ Your friends might ask. ‘The_Girl_who_came_with_John.’ And they will all know. ‘It was nice to meet you girl_who_came_with_john, see you around.’ The good thing is that variables are not really like people. You can be a bit rude to them, they will not mind. You don’t have to force yourself or anyone else to remember an arbitrary name of a variable, that accidentally came to your mind in the moment of creation. Let your colleagues figure out what is what by a meaningful, straightforward description of it.

There is an important tradeoff to be aware of here. The lines of code should not exceed a certain length (79 characters, according to the PEP 8), therefore, it is recommended that you keep your names as short as possible. It is worth to give it a bit of thought about how you can name your variable in the most descriptive way, keeping it as short as possible. Keep in mind, that
the_blond_girl_in_a_dark_blue_dress_who_came_with_John_to_this_party might not be the best choice.

There are a few additional pieces of advice when it comes to naming your variables. First, try to always use pronounceable names. If you’ve ever been to an international party, you will know how much harder to remember is something that you cannot even repeat. Second, you probably have been taught over and over again that whenever you create a loop, you use i and j to denote the iterators.

It is probably engraved deep into the folds in your brain to write for i in…. You need to try and scrape it out of your cortex. Think about what the i stands for, what it really does and name it accordingly. Is i maybe the row_index? Is it a list_element?

Additionally, think about when to use a noun and where a verb. Variables usually are things and functions usually do things. So, it might be better to name functions with verb expressions, for example: get_id() or raise_to_power().

Moreover, it is a good practice to name constant numbers in the code. First, because when you name them you explain the meaning of the number. Second, because maybe one day you will have to change that number. If it appears multiple times in your code, you will avoid searching and changing it in every place. PEP 8 states that the constants should be named with UPPER_CASE_NAME. It is also quite common practice to explain the meaning of the constants with an inline comment at the end of the line, where the number appears. However, this approach will increase the line length and will require repeating the comment if the number appears more than one time in the code.

Emilia Cheladze

Matrix search: Finding the blocks of neighboring fields in a matrix with Python

/0 Comments/in , , /by

Task

In this article we will look at a solution in python to the following grid search task:

Find the biggest block of adjoining elements of the same kind and into how many blocks the matrix is divided. As adjoining blocks, we will consider field touching by the sides and not the corners.

Input data

For the ease of the explanation, we will be looking at a simple 3×4 matrix with elements of three different kinds, 0, 1 and 2 (see above). To test the code, we will simulate data to achieve different matrix sizes and a varied number of element types. It will also allow testing edge cases like, where all elements are the same or all elements are different.

To simulate some test data for later, we can use the numpy randint() method:

The code

How the code works

In summary, the algorithm loops through all fields of the matrix looking for unseen fields that will serve as a starting point for a local exploration of each block of color – the find_blocks() function. The local exploration is done by looking at the neighboring fields and if they are within the same kind, moving to them to explore further fields – the explore_block() function. The fields that have already been seen and counted are stored in the visited list.

find_blocks() function:

  1. Finds a starting point of a new block
  2. Runs a the explore_block() function for local exploration of the block
  3. Appends the size of the explored block
  4. Updates the list of visited points
  5. Returns the result, once all fields of the matrix have been visited.

explore_block() function:

  1. Takes the coordinates of the starting field for a new block and the list of visited points
  2. Creates the queue set with the starting point
  3. Sets the size of the current block (field_count) to 1
  4. Starts a while loop that is executed for as long as the queue is not empty
    1. Takes an element of the queue and uses its coordinates as the current location for further exploration
    2. Adds the current field to the visited list
    3. Explores the neighboring fields and if they belong to the same block, they are added to the queue
    4. The fields are taken off the queue for further exploration one by one until the queue is empty
  5. Returns the field_count of the explored block and the updated list of visited fields

Execute the function

The returned result is biggest block: 4, number of blocks: 4.

Run the test matrices:

Visualization

The matrices for the article were visualized with the seaborn heatmap() method.

Emilia Cheladze

Article series: 5 Clean Coding Tips – 1. Be Consistent

/0 Comments/in , , /by
This is the first of the article series “5 tips for clean coding” to follow as soon as you’ve made the first steps into your coding career, in this article series. Read the introduction here, to find out why it is important to write clean code if you missed it.

Consistency is THE rule to follow if you want to make your code clean and increase readability. Not to make it sound desperate, but honestly, whatever you decide to do when it comes to the coding style, just be consistent. Whether you agree with any standards, formatting styles or don’t even know them, just be consistent. Don’t ever allow inconsistency to sneak into your script or your project. This will only bring confusion, disorientation, chaos and general misery.

The rules for how exactly keep your code clean and organized visually might differ slightly depending on the situation you find yourself in. The PEP 8 rules can be ambiguous in some places and leave room for interpretation. For example, the question, whether you use single or double quotes to denote a string, is open. It is possible, that your work environment already has a standard and you just need to comply with that. No room to show off your highly unique take on it, sorry. However, if you are working on your own and there is no one to roll their eyes looking at your messed-up code, you need to decide for yourself. Once you do, again, be consistent at the level of the script, project, your work in general. Otherwise, it will look messy, patchworky and simply unprofessional.

People famously are quick to ascribe intentionality, even to thermostats[i]. They will assume that the details of how you wrote your code are intentional. They will try to figure out why you are doing one thing in some places and a different thing in other places. If those differences came from you being careless and have no meaning behind them, the reader of your code will waste a lot of time trying to figure it out and end up frustrated. Remember the first few snippets of python code you have ever seen? Maybe you saw some code with double quotes and some with single quotes. You were green, knew nothing and quite possibly thought that they both have different meanings and you spent time trying to figure out why on earth in some places there is a single quote and in other double-quotes.

If those altruistic arguments do not really convince you, let’s see how consistency can serve to your own benefit. First, that outsider, who is looking at your code and is trying very hard to figure out what on Earth is going on, might be you. It might sound crazy, and it is, indeed, quite sad, but most likely, after 6 months of not looking at your code you will no longer remember what you did there if it is not documented well. Documenting in a homogenous way can take some time and some effort. Nevertheless, in general, code gets read many times after it has been written. When in doubt, sacrifice some of your writing time to increase readability and minimize the reading time later. It will pay off in the long run.

Having a set of rules at your disposal can make your work faster. You will avoid arguing with yourself about which option is the best one: mean_income, income_mean or income_avg. You can avoid making loads of small decisions as you write your code by making a set of global rules. In that way, you can allocate your energy and resources into solving the real problem. Not the how-do-I-format-this? one.

It is not necessary that you make all those grand decisions right now. You also don’t have to make them for life, it’s ok to change your mind eventually, so don’t feel overwhelmed. But once you’ve learned this and that, spent a little time coding, have a good long look at your sprouting habits and decide what you are going to do about splitting those lines and stick to it!

References:

[i] https://en.wikipedia.org/wiki/Intentional_stance

Aakash Chugh

Introduction to Recommendation Engines

/0 Comments/in , , , , , , , , /by

This is the second article of article series Getting started with the top eCommerce use cases. If you are interested in reading the first article you can find it here.

What are Recommendation Engines?

Recommendation engines are the automated systems which helps select out similar things whenever a user selects something online. Be it Netflix, Amazon, Spotify, Facebook or YouTube etc. All of these companies are now using some sort of recommendation engine to improve their user experience. A recommendation engine not only helps to predict if a user prefers an item or not but also helps to increase sales, ,helps to understand customer behavior, increase number of registered users and helps a user to do better time management. For instance Netflix will suggest what movie you would want to watch or Amazon will suggest what kind of other products you might want to buy. All the mentioned platforms operates using the same basic algorithm in the background and in this article we are going to discuss the idea behind it.

What are the techniques?

There are two fundamental algorithms that comes into play when there’s a need to generate recommendations. In next section these techniques are discussed in detail.

Content-Based Filtering

The idea behind content based filtering is to analyse a set of features which will provide a similarity between items themselves i.e. between two movies, two products or two songs etc. These set of features once compared gives a similarity score at the end which can be used as a reference for the recommendations.

There are several steps involved to get to this similarity score and the first step is to construct a profile for each item by representing some of the important features of that item. In other terms, this steps requires to define a set of characteristics that are discovered easily. For instance, consider that there’s an article which a user has already read and once you know that this user likes this article you may want to show him recommendations of similar articles. Now, using content based filtering technique you could find the similar articles. The easiest way to do that is to set some features for this article like publisher, genre, author etc. Based on these features similar articles can be recommended to the user (as illustrated in Figure 1). There are three main similarity measures one could use to find the similar articles mentioned below.

 

Figure 1: Content-Based Filtering

 

 

Minkowski distance

Minkowski distance between two variables can be calculated as:

(x,y)= (\sum_{i=1}^{n}{|X_{i} - Y_{i}|^{p}})^{1/p}

 

Cosine Similarity

Cosine similarity between two variables can be calculated as :

\mbox{Cosine Similarity} = \frac{\sum_{i=1}^{n}{x_{i} y_{i}}} {\sqrt{\sum_{i=1}^{n}{x_{i}^{2}}} \sqrt{\sum_{i=1}^{n}{y_{i}^{2}}}} \

 

Jaccard Similarity

 

J(X,Y) = |X ∩ Y| / |X ∪ Y|

 

These measures can be used to create a matrix which will give you the similarity between each movie and then a function can be defined to return the top 10 similar articles.

 

Collaborative filtering

This filtering method focuses on finding how similar two users or two products are by analyzing user behavior or preferences rather than focusing on the content of the items. For instance consider that there are three users A,B and C.  We want to recommend some movies to user A, our first approach would be to find similar users and compare which movies user A has not yet watched and recommend those movies to user A.  This approach where we try to find similar users is called as User-User Collaborative Filtering.  

The other approach that could be used here is when you try to find similar movies based on the ratings given by others, this type is called as Item-Item Collaborative Filtering. The research shows that item-item collaborative filtering works better than user-user collaborative filtering as user behavior is really dynamic and changes over time. Also, there are a lot more users and increasing everyday but on the other side item characteristics remains the same. To calculate the similarities we can use Cosine distance.

 

Figure 2: Collaborative Filtering

 

Recently some companies have started to take advantage of both content based and collaborative filtering techniques to make a hybrid recommendation engine. The results from both models are combined into one hybrid model which provides more accurate recommendations. Five steps are involved to make a recommendation engine work which are collection of data, storing of data, analyzing the data, filtering the data and providing recommendations. There are a lot of attributes that are involved in order to collect user data including browsing history, page views, search logs, order history, marketing channel touch points etc. which requires a strong data architecture.  The collection of data is pretty straightforward but it can be overwhelming to analyze this amount of data. Storing this data could get tricky on the other hand as you need a scalable database for this kind of data. With the rise of graph databases this area is also improving for many use cases including recommendation engines. Graph databases like Neo4j can also help to analyze and find similar users and relationship among them. Analyzing the data can be carried in different ways, depending on how strong and scalable your architecture you can run real time, batch or near real time analysis. The fourth step involves the filtering of the data and here you can use any of the above mentioned approach to find similarities to finally provide the recommendations.

Having a good recommendation engine can be time consuming initially but it is definitely beneficial in the longer run. It not only helps to generate revenue but also helps to to improve your product catalog and customer service.

Deep Moteria

Python vs R: Which Language to Choose for Deep Learning?

/1 Comment/in , , , , /by

Data science is increasingly becoming essential for every business to operate efficiently in this modern world. This influences the processes composed together to obtain the required outputs for clients. While machine learning and deep learning sit at the core of data science, the concepts of deep learning become essential to understand as it can help increase the accuracy of final outputs. And when it comes to data science, R and Python are the most popular programming languages used to instruct the machines.

Python and R: Primary Languages Used for Deep Learning

Deep learning and machine learning differentiate based on the input data type they use. While machine learning depends upon the structured data, deep learning uses neural networks to store and process the data during the learning. Deep learning can be described as the subset of machine learning, where the data to be processed is defined in another structure than a normal one.

R is developed specifically to support the concepts and implementation of data science and hence, the support provided by this language is incredible as writing codes become much easier with its simple syntax.

Python is already much popular programming language that can serve more than one development niche without straining even for a bit. The implementation of Python for programming machine learning algorithms is very much popular and the results provided are accurate and faster than any other language. (C or Java). And because of its extended support for data science concept implementation, it becomes a tough competitor for R.

However, if we compare the charts of popularity, Python is obviously more popular among data scientists and developers because of its versatility and easier usage during algorithm implementation. However, R outruns Python when it comes to the packages offered to developers specifically expertise in R over Python. Therefore, to conclude which one of them is the best, let’s take an overview of the features and limits offered by both languages.

Python

Python was first introduced by Guido Van Rossum who developed it as the successor of ABC programming language. Python puts white space at the center while increasing the readability of the developed code. It is a general-purpose programming language that simply extends support for various development needs.

The packages of Python includes support for web development, software development, GUI (Graphical User Interface) development and machine learning also. Using these packages and putting the best development skills forward, excellent solutions can be developed. According to Stackoverflow, Python ranks at the fourth position as the most popular programming language among developers.

Benefits for performing enhanced deep learning using Python are:

  • Concise and Readable Code
  • Extended Support from Large Community of Developers
  • Open-source Programming Language
  • Encourages Collaborative Coding
  • Suitable for small and large-scale products

The latest and stable version of Python has been released as Python 3.8.0 on 14th October 2019. Developing a software solution using Python becomes much easier as the extended support offered through the packages drives better development and answers every need.

R

R is a language specifically used for the development of statistical software and for statistical data analysis. The primary user base of R contains statisticians and data scientists who are analyzing data. Supported by R Foundation for statistical computing, this language is not suitable for the development of websites or applications. R is also an open-source environment that can be used for mining excessive and large amounts of data.

R programming language focuses on the output generation but not the speed. The execution speed of programs written in R is comparatively lesser as producing required outputs is the aim not the speed of the process. To use R in any development or mining tasks, it is required to install its operating system specific binary version before coding to run the program directly into the command line.

R also has its own development environment designed and named RStudio. R also involves several libraries that help in crafting efficient programs to execute mining tasks on the provided data.

The benefits offered by R are pretty common and similar to what Python has to offer:

  • Open-source programming language
  • Supports all operating systems
  • Supports extensions
  • R can be integrated with many of the languages
  • Extended Support for Visual Data Mining

Although R ranks at the 17th position in Stackoverflow’s most popular programming language list, the support offered by this language has no match. After all, the R language is developed by statisticians for statisticians!

Python vs R: Should They be Really Compared?

Even when provided with the best technical support and efficient tools, a developer will not be able to provide quality outputs if he/she doesn’t possess the required skills. The point here is, technical skills rank higher than the resources provided. A comparison of these two programming languages is not advisable as they both hold their own set of advantages. However, the developers considering to use both together are less but they obtain maximum benefit from the process.

Both these languages have some features in common. For example, if a representative comes asking you if you lend technical support for developing an uber clone, you are directly going to decline as Python and R both do not support mobile app development. To benefit the most and develop excellent solutions using both these programming languages, it is advisable to stop comparing and start collaborating!

R and Python: How to Fit Both In a Single Program

Anticipating the future needs of the development industry, there has been a significant development to combine these both excellent programming languages into one. Now, there are two approaches to performing this: either we include R script into Python code or vice versa.

Using the available interfaces, packages and extended support from Python we can include R script into the code and enhance the productivity of Python code. Availability of PypeR, pyRserve and more resources helps run these two programming languages efficiently while efficiently performing the background work.

Either way, using the developed functions and packages made available for integrating Python in R are also effective at providing better results. Available R packages like rJython, rPython, reticulate, PythonInR and more, integrating Python into R language is very easy.

Therefore, using the development skills at their best and maximizing the use of such amazing resources, Python and R can be togetherly used to enhance end results and provide accurate deep learning support.

Conclusion

Python and R both are great in their own names and own places. However, because of the wide applications of Python in almost every operation, the annual packages offered to Python developers are less than the developers skilled in using R. However, this doesn’t justify the usability of R. The ultimate decision of choosing between these two languages depends upon the data scientists or developers and their mining requirements.

And if a developer or data scientist decides to develop skills for both- Python and R-based development, it turns out to be beneficial in the near future. Choosing any one or both to use in your project depends on the project requirements and expert support on hand.