Batch processing large text files for sentiment analysis

I understand you are a beginner and you had the guts to post your code. You deserve some feedback and I will try.

Your code is hard to read

This is most probably why you didn't get answers till now. There is code repetition, global and reused variables, no clear control flow, and bad variable names. It is really hard to find the core functionality.
So I start in no particular order.

Bad names

dic = {}                                #dictionary for storing list elements

Well I can see this is a dict. Bad name, bad comment. Use names describing the meaning, not the type. Comments shoud describe the hidden, not so obvious parts, that could not be expressed by names. Your whole comment #dictionary for storing list elements can be split into  #dictionary for storing ... elements and ... lists ....
the first one is completely obvious, the other is just some type information again. No hints to the sort of content (should be in the name), no hints as to the key.
You could safely replace the whole line with guesswhat={}.

Do not use globals or nonlocals for no reason

def for_high (list_e):                  #defining a function for taking list elements and putting them into dic

Uses filename, res, and l. While one could expect that on res and as it was declared at global scope, it is a complete no-go to use variables that look like temporaries in their scope. This renders your function absolutely untestable and your function will easily break if you alter the outer code. Pass required variables, so the reader/maintainer is aware they are used by the function.
Unlike what the comment suggests, your function currently only takes a list length. And unless for_high is a well known / defined term in your community, it is a bad name again.

Do not repeat yourself (DNRY)

Your function has some code:

output = paralleldots.emotion(dic.get(x))
res.update({x: output})
with open(os.path.join("C:\\Users\\User\\Desktop\\trial", filename), 'w') as text_file:
    text_file.write(json.dumps(res))

which is duplicated outside (slightly modified)

in_txt = paralleldots.emotion(l)
filename = os.path.splitext(os.path.basename(txt_file))[0] + '.txt'
with open(os.path.join("C:\\Users\\User\\Desktop\\trial", filename), 'w') as text_file:
    text_file.write(str(in_txt))

How can something be either output or in_txt? Of course a function has a return value or output or result. None of these make a good name. What is returned?
For one code snippet filename is explicitly set, for the other it magically pre-exists? So the function accesses a value from outside and the outer code overwrites the existing value with the very same value before using it.
This is pure horror. Also one of this file writes is done in a loop, the file is overwritten in each iteration. It shall either be written once or appended.
Create a function handling one element.

Simplify control flow

If both execution paths of an if clause nearly do the same, check whether you can simplify it. Your

if list_e % 2 != 0:             #checking if list has an odd no. of elements
    list_e += 1
    for_high(list_e)

else:
    for_high(list_e)

can be simplified to (again add a useful comment):

if list_e % 2 != 0:             # list length may not be odd because ...
    list_e += 1
for_high(list_e)

but: are you sure it is a good idea to increase a list-length variable while the list itself stays odd?

Define numeric constants once

you have

no_keys = list_e / 20
dic = {1: l[1:21]}
start_key_value = 21
end_key_value = start_key_value + 20
start_key_value += 20
if list_e > 20:                     #checking if list has more than 20 elements

I do not think these are independent constants. So define this constant once and use that name throughout the code (again change the comment to have some meaning)

max_emoticon_len = 20
no_keys = list_e / max_emoticon_len
dic = {1: l[1:max_emoticon_len + 1]}
start_key_value = max_emoticon_len + 1
end_key_value = start_key_value + max_emoticon_len
start_key_value += max_emoticon_len
if list_e > max_emoticon_len:       #checking if list has more than elements than desired to fit for ...

Other

Also list_e is a bad name. e mostly is used as name/pre-/postfix for elements for e in l:. for no. of elements the term length as in len() is used. stick to those conventions to avoid irritation.

Also dic = {1: l[1:21]} - are you sure you want to skip the first element? In Python, indexing starts at 0.

Also is it desired, that when having a list of length 40, the first sequence has length 20, and the last one has length 19?

Also dic = {1: l[1:21]} hides the outer dic which therefor is never used and may be deleted

Also dic = {1: l[1:21]} - why do you set this explicitly? Why not inside the loop?

Also dic.update({i: 'some'}) may be done by dic[i] = 'some'. The same goes for res.

Also you handle lists with less than 20 elements in your main code. However, if the list is bigger than 20 elements, you do batches of 20 and drop the last ones. Desired?


Non-repeatability (plagiarism and time travel)

More classes

Making the author guesser a class was a good start, but I think authors also should be a class.  Making them a class now will allow you to work more with the concept of an author in the future (see "More work to do").

Cleaning up

You don't really need the book reading method, and you can simplify how you teach your guesser about authors and then test them later.  I've tried to leave the general spirit of your code intact:

from collections import Counter
from string import ascii_letters, whitespace

def filter_words(text):
    '''removes weird characters and split into words'''
    remain = ascii_letters + whitespace
    filtered = [ch.lower() for ch in text if ch in remain]
    return "".join(filtered).split()

def most_used(words, n=500):
    '''returns a set of the "n" most used words'''
    most_used_words = Counter(words).most_common(n)
    return set([word for word, frequency in most_used_words])

def read_and_analyse(filename):
    '''reads a book and returns most used words'''
    with open(filename) as book:
        words = filter_words(book)
        favorites = most_used(words)
        return favorites

class Author(object):
    def __init__(self, name, favorite_words):
        self.name = name
        self.favorite_words = favorite_words

class WriterGuesser(object):
    def __init__(self):
        self.analyzed_authors = []

    def learn_about_authors(self, book, name):
        favorite_words = read_and_analyse(book)
        author = Author(name, favorite_words)
        self.analyzed_authors.append(author)
        print "I read {} by {}".format(book, name)

    def recognize_author(self, book):
        favorite_words = read_and_analyse(book)
        max_identical_words = 0
        guess = None
        for author in self.analyzed_authors:
            common = len(favorite_words.intersection(author.favorite_words))
            if common > max_identical_words:
                max_identical_words = common
                guess = author.name
        print"I think {} is written by {} ({} words in common)".format(book, guess, max_identical_words)

def main():
    nerd = WriterGuesser()

    learning_criteria = [
        ('WutheringHeights.txt', 'Bronte'),
        ('PrideAndPrejudice.txt', 'Austen'),
        ('TheMerchantOfVenice.txt', 'Shakespeare')
    ]

    for book, author in learning_criteria:
        nerd.learn_about_authors(book, author)

    books_to_recognize = [
        'SensAndSensibility.txt',
        'RomeoAndJuliet.txt',
        'JaneEyre.txt',
    ]

    for book in books_to_recognize:
        nerd.recognize_author(book)

if __name__ == "__main__":
    main()

More work to do

You can only learn about one book from an author.  Surely, you'd like a system that allows you to create an author and then use several books to develop their favorite words.  This is where a class is helpful - you can create the author class, get their favorite words, and then update them as you learn more books.
You should expand this so that you can use many books by the same author to learn favorite words.  Perhaps return the frequency of a word along with the word itself (which Counter already does), and remove less popular words as more popular ones come in.
You should also use the frequency of the word to determine how confident you are that the authors are the same.  Say an author's favorite words are ["cat", "purple", "clock"], but they use "clock" 600 times and use the rest only 10 times.  Your code will pick a book with "cat" and "purple" over a book with 500 instances of "clock", because 2 matching words > 1 matching word.  But we know "clock" is a more important word!
Everybody's favorite words are going to be "the", "it", "a", "and", etc.  Consider making a list of boring words and not using them for your filtering.


There are a lot of magic numbers.  They are numbers that would be familiar to any good programmer, but magic numbers nonetheless.  Furthermore, it's more useful to think of the problem using bitwise operations, since they are more closely related to the conceptual design behind UTF-8.

You only accept two-, three-, and four-byte sequences, i.e., only code points U+0080 to U+1FFFFF, i.e., the Basic Multilingual Plane (minus the ASCII set) plus the Supplementary Multilingual Plane.  That should be made clear in comments.

I suggest renaming IS_UTF8_BYTE() to IS_UTF8_MULTIBYTE(), since ASCII characters ≤ 127 are valid UTF-8 bytes too — just not members of a multibyte sequence.

//check if value is in range of leading byte (0b11?????? but not 0b11111???)
#define IS_UTF8_LEADING_BYTE(b) (((b) & 0xc0) && !((b) & 0xf8))

//check if value is in range of sequence byte (0b10??????)
#define IS_UTF8_SEQUENCE_BYTE(b) (((b) & 0x80) && !((b) & 0xc0))

//can be any byte within a UTF-8 multibyte sequence
#define IS_UTF8_MULTIBYTE(b) (IS_UTF8_LEADING_BYTE(b) || IS_UTF8_SEQUENCE_BYTE(b))

//no error checking; it must be used only on leading byte
#define HOW_MANY_UTF8_SEQUENCE_BYTES(b) (((b) & 0x40 == 0x40) + ((b) & 0x20 == 0x20) + ((b) & 0x10 == 0x10))

IS_UTF8_SEQUENCE_BYTE() could be more clever.

//check if value is in range of sequence byte (0b10??????)
#define IS_UTF8_SEQUENCE_BYTE(b) ((b) & 0xc0 == 0x80)


Function for accessing and searching dictionaries using /slashed/paths (like xpath)

I would skip the [] in the path.  Just presume val is a dict with strings for keys (the usual case).  It that doesn't work, try converting the key to an int.
def dict_query(d: Dict, path: str) -> Union[object, None]:
    """
    # Example usage:
    >>> d = {'a': [{}, {'b': 9}]}
    >>> print( dict_query(d, 'a/1/b') )
    9
    """
    keys = path.split("/")
    val = d

    try:
        for key in keys:
            try:
                val = val[key]

            except (KeyError, TypeError):
                val = val[int(key)]

        return val

    except (IndexError, KeyError, TypeError, ValueError):
        return None


General
Your code formatting is all over the place. You should use consistent and structured indentation. See the Java Code-Style guide for, well, guidance.
if I take your code, and re-format it (Eclipse, CtrlA and CtrlshiftF), it looks like:

public static String removeDuplicate(String s) {
    StringBuilder builder = new StringBuilder();
    char lastchar = '\0';
    for (int i = 0; i < s.length(); i++) {
        String str = builder.toString();
        if (!str.equals("") && (str.charAt(str.length() - 1) == s.charAt(i))) {
            builder.deleteCharAt(str.length() - 1);
        } else if (s.charAt(i) != lastchar)
            builder.append(s.charAt(i));
        lastchar = s.charAt(i);
    }
    return builder.toString();
}

At least this allows us to see what you are doing.
Algorithm
A nice algorithm will take data from the input string, and add it to the output if it should be added. A system where you do multiple conversions, add things, and remove things, is complicated, and hard to follow.
Additionally, the null character \0 is actually a valid character in Java, so you may have (an extremely rare) bug.
I do not like that your code is removing all the duplicate values, it seems more practical to remove all but one of the duplicates, but you do not explain why it is supposed to be this way.
More Information required
The above, in itself, is a review, but, if you update your question with the requested details:

recursion yes/no/why?
dedup all, or all-but-one?

then I can add suggestions as to how to solve this in a more efficient way.


A Scala Maze Generator in Functional Style

Prefer Classes to Multiple Tuples

There are lots of usages of cell: (Int, Int), with multiple occurrences of cell._1 and cell._2. Tuples are nice and suitable for various situations, but when there are too many of them, it reduces readability and increases risks of bugs. And you need to remember that _1 is x and _2 is y. Or vice versa. Or both.

So a simple way to improve it is just create an ad-hoc class

private class Cell(val x: Int, val y: Int)

It might also be a case class, if necessary.

Use Options

The method signature

private def getRandomUnvisitedNeighbor(maze: Array[Array[Int]], cell: (Int, Int)): (Boolean, (Int, Int))

is suspicious.

Not only it returns a tuple containing another tuple, but also it looks like the purpose of the Boolean is to show if the return object contains data or marks a missing value, according to mazify: if (unvisitedNeighbor._1) { ... }.

Scala has a dedicated Option[T] type for that:

private def getRandomUnvisitedNeighbor(maze: Array[Array[Int]], cell: (Int, Int)): Option[(Int, Int)] = {
  val neighbors = getUnvisitedNeighbors(maze, cell)
  if (neighbors.length > 0)
    Some(neighbors(Random.nextInt(neighbors.size))))
  else
    None
}

In mazify:

...
getRandomUnvisitedNeighbor(maz, currentCell) match {
  case cell: Some[(Int, Int)] => {
    val newCurrentCell = cell.get
    maz(newCurrentCell._1)(newCurrentCell._2) = 1
    mazify(currentCell :: stack, newCurrentCell)
  }
  case None => {
    val newCurrentCell = stack.head
    mazify(stack.tail, newCurrentCell)
  }
}

Nested fors

The nested for loops in Maze.toString look too imperative-style.

The nesting can be avoided with

for {
  i <- 0 until height
  j <- 0 until length
} {
  strBuilder.append(maz(i)(j))
  if (j == length - 1) strBuilder.append("\n")
}

  @jwill's question 2. What is the benefit of the for comprehensions? It looks like the same mutability as a double for loop.

Yes, there is no magic and it is equivalent to double/nested for loops.
As I already mentioned, when we have

for (bla-bla-bla_1) {
  for (bla-bla-bla_2) {
    // very clean code here
  }
}

It looks like the good/bad old imperative style. In Scala, foreach and others are functional style replacements for traditional for. The for-comprehension is a sort of shortcut allowing to iterate, filter and map (with yield). For example:

for {
  i <- 0 until height
  if i % 2 == 0
  j <- 0 until length
  if j % 2 == 0
} yield {
  (i, j)
}

Will produce tuples of even values of i and j.
It is completely equivalent to

(0 until height).filter(_ % 2 == 0).flatMap(i => {
   (0 until length).filter(_ % 2 == 0).map(j => (i, j))
})

But clearly more readable with for. And the nesting is not visible.

  @jwill's question 5. Is the stringbuilder frowned upon?

StringBuilder (either from java.lang or scala.collection) is THE object to use when you need to dynamically construct a String from numerous concatenations.

Readability

getUnvisitedNeighbors contains a terribly complex filtering condition and unnecessary vals. Slightly changed:

private def getUnvisitedNeighbors(maze: Array[Array[Int]], cell: (Int, Int)) = {
  def xInBounds(x: Int) = x >= 0 && x < length
  def yInBounds(y: Int) = y >= 0 && y < height
  neighbors(cell).filter(c => {
    xInBounds(c._2) &&
    yInBounds(c._1) &&
    maze(c._1)(c._2) != 1 &&
    hasValidNeighbors(c, cell)
  })
}

  @jwill's question 3. In getUnvisitedNeighbors, why did you write those intermediary calculations as defs rather than vals?

Briefly speaking, because def is a function that is invoked when necessary. val is a constant reference, calculated only once on first invocation. If vals were used here instead of defs, we would not be able to check the condition from within the filter method for each particular cell.

  @jwill's question 4. You also didn't mention anything about nesting the private defs inside the main override def maze function. Is that important at all, or a scala idiom to do so?

If you mean that def maze invokes private def mazify, that is absolutely OK and mazify should remain as a private function. Nesting defs within each other should not be overused. I would recommend it only when the nested function is 1) really short; 2) simplifies the readability of the parent one; 3) never needs to be invoked from other functions or itself; 4) does not call other functions of this entity. For def mazify most these conditions are not met.

Mutability

  @jwill's question 1. You didn't mention any adjustments for the mutations of the maze. How come? Does it follow under the principle of, "Keep your mutations in small, select locations"?

Indeed, I didn't cover it in the initial answer.

var is not necessary for maz field, it can and should be transformed to val. The reference is never changed throughout the code, but only the contents of the array changes. And I also think that there is no need to expose the array with public access at all.

The def maze in DepthFirstSearch also seems disturbing, because it recalculates the maze on each call and returns a new different instance of DepthFirstSearch. Was it really the intention?

It would be more logical to remove both var maz and def maze from trait Maze and define an accessor for cell values at the trait level:

trait Maze {
  val height: Int
  val length: Int
  def valueAtCell(x: Int, y: Int): Int
}

The implementing class will encapsulate the Array with cells values:

class DepthFirstSearch(override val height: Int,
                       override val length: Int) extends Maze {

  private val maze: Array[Array[Int]] = {
    val maz = Array.ofDim[Int](height, length)
    maz(0)(0) = 1
    mazify(List(), (0, 0))
    maz
  }

  override def valueAtCell(x: Int, y: Int): Int = maze(y)(x)

  private def mazify(stack: List[(Int, Int)], currentCell: (Int, Int)): Unit = {
    val unvisitedNeighbor = getRandomUnvisitedNeighbor(maze, currentCell)
    if (unvisitedNeighbor._1) {
      val newCurrentCell = unvisitedNeighbor._2
      maze(newCurrentCell._1)(newCurrentCell._2) = 1
      mazify(currentCell :: stack, newCurrentCell)
    } else if (stack.nonEmpty) {
      val newCurrentCell = stack.head
      mazify(stack.tail, newCurrentCell)
    }
  }
...
}


map and reduce are tailor made for problems like this.

To understand the while loop in the final version of the code at the bottom of this post, it may help to consider a "longhand" version that does the same thing:

// this sets the answer we seek to the sum of all the digits
// of every number passed in via the array
finalAnswer = addDigits(combinedDigits);

// but since this sum may still be larger than a 1 digit
// number (ie, 10 or more), we need to continue the summing
// the digits until we end up with a single digit number
while ( finalAnswer > 9 ) {
  // this converts an integer to a string
  combinedDigits = '' + finalAnswer;
  // this sums the digits again
  finalAnswer = addDigits(combinedDigits);
}

Here's the final, complete version, with the more compact while loop.  We are simply combining some of the steps above.

function addDigitsRecursively(arr) {

  var combinedDigits = arr.join(''), finalAnswer;

  while ( (finalAnswer = addDigits(combinedDigits)) > 9 )
    combinedDigits = '' + finalAnswer;

  function addDigits(str) {
    return str.split('').map(function(n) {return parseInt(n)})
                        .reduce(function(a, b){return a+b;})
  }

  return finalAnswer;
}

var value = addDigitsRecursively([13, 11, 1938]);
document.body.innerHTML = value;


How can I improve the isolation for this controller spec?

It's fine to test how controllers alter models.


Calculate hat, waist, and jacket size when given age, height, and weight

I see some things that may help you improve your code.

Don't abuse using namespace std

Putting using namespace std at the top of every program is a bad habit that you'd do well to avoid.  An alternative that may still meet your needs may be to put it within main rather in the global context.

Eliminate parentheses to enhance readability

Adding parentheses where they are not needed makes the code harder to read and comprehend than it needs to be.  For example, in the hat function, we have this:

return ((weight/height) * 2.9);

With no change in the effective calculation, we can simply write this:

return weight / height * 2.9;

I think that's a little easier to read.

Fix the formatting

It seems you may already know this, but your indentation and use of whitespace is not consistent.  It's less important which particular conventions you use than choosing one and applying it consistently.

Specify parameter names in function prototypes

With a function prototype like this:

double hat(double,double);

I can't tell which parameter is the height and which is the weight without looking at the implementation code.  Since function prototypes tend to become the documentation for your code's public interface, make it easier on the user of these functions by writing the prototype like this:

double hat(double weight, double height);

Or even like this to make the units unambiguous:

double hat(double weightInPounds, double heightInInches);

Don't pass parameters that are not needed

The waist routine does not need or use the height, so that parameter can and should be eliminated.

Use all required #includes

The code uses toupper() but doesn't include the corresponding header.  The code should have

#include <cctype>

Simplify expressions

The jacket and waist calculations are more complex than they need to be.  For example, for the waist routine, one could write this:

double waist(double weight, int age)
{
    double waist = weight / 5.7;
    if (age > 28) {
        waist += (age - 28) / 2 * 0.1;
    }
    return waist;
}

Note that the calculation (age - 28) / 2 is done using integer math, so that subexpression will yield 0 for ages of 28 or 29 as per the specification.  Understanding implicit conversions and particularly numeric conversions will be very helpful to you as you write future programs.


Since you tagged this question c++ I assume you would like a C++ approach also.

I would suggest using std::vector<int> rather than an int* and malloc. Using the vector will give you a number of useful advantages over a normal pointer, for example, memory management is greatly simplified, and there is no need to worry about how big an int is. With vector, the order of all elements is guaranteed to be contiguous.

#include <algorithm>
#include <functional>
#include <iostream>
#include <vector>

int nextNumber(int k, const std::vector<int> &array, int n)
{
    std::vector<int>::iterator i = std::find_if(array.begin(), array.begin() + n, std::bind2nd(std::greater<int>(), k));

    return (i == array.end() ? -1 : *i);
}

bool isMultiple(int a, int b)
{
    return (a % k == 0) && (a != k);
}

void markMultiples(int k, std::vector<int> &array, int n)
{
    std::replace_if(array.begin(), array.begin() + n, std::bind2nd(std::ptr_fun(isMultiple), k), -1);
}

void printArray(const std::vector<int> &array)
{
    for(int i = 0; i < array.size(); i++)
        std::cout << "array[" << i << "] = " << array[i] << std::endl;
}

Disclaimer:

I am not very good at C++, so you may want to take this advice with a teaspoon of salt.


Scraping information from a webpage not knowing it's last page number

I tried to make it error-free.

Except this is the Web, and whether a server located somewhere on the globe will respond with something you expect, is completely out of your control.

This instruction:

.Open "GET", mlink & y, False

Succeeds under normal circumstances, but one day the site will be down for maintenance, or whatever - and you'll get a run-time error, be it here or at the .Send call.

Code that doesn't handle errors is code that is written for the "happy path" - it's code that works well, until it doesn't. And then, when one thing goes wrong, everything burst up in flames, in a quite inelegant way. There are ways to graciously handle run-time errors.

Use them.

On Error GoTo CleanFail

Looks like you like scraping stuff. That's great, but at one point you need to solve the more generic problem, and move the URL from a local Const to a parameter.

Consider implementing the website-specifics as interface implementations.

@Interface
Option Explicit

Public Function Parse(ByVal url As String) As VBA.Collection
End Function

And when you start implementing interfaces in VBA, you'll find that VBA will refuse to compile when you have an implemented public interface member with an underscore in its name - so you might as well drop that habit now.

The indentation is wrong. Get the latest Rubberduck and use its Smart Indenter. Rubberduck will also warn you about other things, e.g. multiple declarations in a single instruction, single-letter identifiers, and As New, which literally makes an indestructible object - which is usually a very bad idea. Try adding this instruction:

Set http = Nothing
Debug.Print http Is Nothing

If you expected that to print True, you've fallen prey to the As New "feature". Best stay away from that when you want to control your objects' life time.


There is an answer in here I hope, and it is a polymorphic perspective. Your description is very reminiscent of a problem where using IEqualityComparer simplified the hell out of things.

Ask yourself if each enum member represents a different class or perhaps groups of enum members represent states for given classes. ... and/or ... Where's the context in such an abstract array? It's hidden in a morass of casees.

Does the length and/or value of certain element(s) determine the "type?" If so then a three step process will break apart the existing code, simplify, and clarify. 1) Evaluate for the type, 2) instantiate via a factory 3) do whatever with the instance.

Create a class that will hold the array (whole, intact) plus appropriate key fields (values extracted from the array). A super-set of key fields so any byte array "type" can be held there. Add a field for the type itself (likely an enum), as distinct from your Code enum.  This class will take an IEqualityComparer object in its constructor.

I had a case where seven distinct "types" were clearly in two groups (per data meaning/use by the users). It was convenient and practical to design only 2 IEqualityComparer classes.

Create "Processor" classes as needed that do stuff with and to these "byte[]-EqualtityComparer" instances individually and in collections.

If you need collections I strongly uge you to create type-specific collections (2 in my case) that by design know how to use the class. In my case I was looking for duplicates.

edit

  Sorry but I couldn't understand how IEqualityComparer come into place here. What do you compare with? Could you elaborate on it? Thank you!

We are lying the foundation for code organization and extensibility. With 20+ "types" this differentiation is also the single responsibility principle in action. This is fundamental functioning that every class/type needs. Now I can identify a plain-old byte[] by it's "type" and it behaves like a unique individual via it's values - as used in Equals, CompareTo, Contains, Sort, Find, etc. w/in the .NET framework.

The entire switch statement will devolve to one line:

ByteArrayClass myBAobject = myFactory.Create( Code.A );

This encapsulates everything that new object can do, now and in the future. I'm assuming it will need Equals() and/or CompareTo()

  I guess my main concern is that I may have too many implementation classes if I use one for each case (except for the common ones which is about 10-15% of all cases

Generic Handling: We can have a single ByteArrayClass collection that can be passed anywhere and used either generally or in type-specific ways.
A general Processing Class. For now: For expediency dump all that code into a ByteArrayClassProcessor class. The code should be organized by "type". This will facilitate refactoring to "type"-specific classes if this processor gets to be too much.
Determing Type: Yeah, the initial grunt work for determining the byte[]'s type must be done. But do the minimum required to do it. From that instantiate a ByteArrayClass object. From then on functionality is type-specific, and most of that switch case comes over from the dark side.

Use the factory pattern.

Validate a "typed" object, not a generic byte[]

I wonder if the byte[] length is type specific, even if it's the same for many, most of the types.

Constantly evaluate code evolution to decide when/if to create new classes

At this point, functionally oriented; "determine type", validation, database interaction, etc.
Create type-specific classes on an as-needed basis

Does not have to be all or nothing.
Use the factory pattern. And in an OMG moment you'll see the potential for this new factory utilizing the "determine type" factory above.
The ByteArrayClass object (output from the "determine type" factory) is wrapped by the new type-specific class.

endEdit


animate button presses in a sequence (tkinter)

Let's look at this code:

for i in self.cpu_moves:
    root.after(200, animate(self.btns[i]))

First, you're not using after correctly. You're doing the equivalent of this:

for i in self.cpu_moves:
    result = animate(self.btns[i])
    root.after(200, result)

You might as well not call root.after at all, since result will be None. If you need to pass parameters to the function you can do so by adding arguments to after itself:

root.after(200, animate, self.btns[i])

Unfortunately, you can only do this with positional parameters, not named parameters. You still need to use lambda the way you are doing it in animate.

Second, this loop is going to end up calling every function at the same time - the first one at "now+200ms", the next one at "now+200ms", the third at "now+200ms" and so on. What you want is the first one to be at "now+200ms", the next at "now+400ms" and so on. A simple solution is to add 200 for each iteration of the loop:

delay=200
for i in self.cpu_moves:
    root.after(delay, animate(self.btns[i]))
    delay += 200


Bugs

A quick bug: if (!started && isspace(c)); has no effect.  Did you want continue; or something there?  As it is, you will store leading spaces in the buffer after all, which apparently is not what you want.
gcc -Wextra gives a warning, btw.
(Did you test the program on input lines with leading spaces?  I don't think it could ever have worked.)
And you currently have no way for started to ever become true...

Code quality improvements

mygets is not a great choice for your function name. A reader will incorrectly guess that it behaves like the standard fgets (or worse, the formerly standard gets), but it differs in that it (is supposed to) strip leading spaces.  I'd choose something more like get_film_title.  You could try to explain in the function name that it strips spaces but that may make it awkwardly long.

I'd split the three tasks of populating the list, printing it out, and freeing it, into three separate functions, instead of dumping them all in main.

The while (true) idiom with a break; makes the reader have to hunt through the loop body for the termination condition.  See if you can think of a clean way to restructure to avoid this.  If not, at least add a comment to explain when the loop will terminate.

See if you can restructure the if (!head) to avoid writing out the malloc call twice, which is a danger point if you eventually want to change it but miss updating both instances.

Some would suggest malloc(sizeof *current) instead of sizeof(struct film).  I don't have strong feelings either way, myself.

Any time you need a comment to explain a "trick", where the comment is longer than the extra code it avoids having to type, you are probably trying to be too clever. (And I found it confusing even with the comment.) Just write out the two statements.

Consider moving the task of reading the rating and "cleaning up the input stream" into its own function.  It's awkward when some of the I/O work is farmed out to a subroutine (mygets) and some is left behind.

for (int c; ...): A for loop that doesn't need to do anything in either of the init-clause or the iteration-expression should probably just be a while.

In the printing code, it's nice to keep the for focused on the actual loop iterator, which here is current, rather than mixing it between current and i (and since the latter is not the iterator, it could use a different name).  And there's no need to bury the increment of i inside a statement that does something else.   I would write:

int film_number = 1;
for (current = head; current != NULL; current = current->next) {
    printf("...", film_number, current->title, current->rating);
    film_number++;
}

In the freeing code, why do you declare a new variable named current, shadowing the existing one?

The for loop and use of the variable next in the freeing code is maybe more clever than readable.  I would write more simply:

current = head;
while (current) {
    struct film *tmp = current;
    current = current->next;
    free(tmp);
}

I think your plan of freeing the list from beginning to end is fine.  With a single-linked list, it's not so easy to do it the other way.  (Unless you want to build a double-linked list instead.

Error checking on every input and memory allocation function.  Yes, it's boring, but essential for any real program, and even for toy programs it is very helpful for debugging.

Suggested enhancements

As to additional metadata: store only what you have a use for.  If  knowing the size or the last element becomes useful for your application, add it then.  But I don't suggest overdesigning a fully general and heavily featureful linked list for a specific program; if you want to reinvent a generic linked list container, that's a separate project (and one that might be better for a language that actually supports generic programming).

Using scanf to read user input is not ideal outside toy programs, as it makes it hard to detect and recover from improper input.  A better design is usually to read an entire input line and parse it out with sscanf.

Consider using dynamic allocation so as to allow the film titles to be of arbitrary length.  Otherwise the user who wants to rate The Englishman Who Went Up a Hill But Came Down a Mountain (57) is going to be peeved.

Command line programs that prompt for interactive input, and terminate on something like an empty line, are rare outside of toy examples.  For one thing, they are hard to adapt for batch processing or pipelines.  Useful command-line programs more often don't prompt, and read input until end-of-file is encountered.


Convert FLAC files to MP3 in parallel on Ubuntu

shellcheck

It is always a good plan to run your code through shellcheck.net.  This only reveals a couple of issues:

Your -j$(nproc) should be quoted to avoid word splitting.  Hopefully no one will put spaces in nproc, but just in case -j"$(nproc)" would handle it better.  As noted in a comment above you may be able to drop this totally
Your {.}.mp3 bothers shellcheck which is also correctable with quoting like '{.}.mp3'.

ifs

I really like that you're using the direct exit value checking with your ifs.  This is a great shell feature that is not terribly popular.

I wish you would indent them a bit better though:

if ! dpkg-query -W -f='${Status}' ffmpeg | grep "ok installed"; then
    apt-get update  && apt-get install ffmpeg;
fi

Lining up the if and the fi makes me feel better.  Having the conditional things indented also helps following along much easier.

shorter line length is good

You're already using \ to split up the long parallel line, so why not split the ffmpeg part of it in two so it all fits in 80 columns?

efficiency

Using parallel to manage the process seems like a good plan.  This operation should be CPU bound so it will do a good job for getting this done.  You might want to have some CPU and I/O monitoring going to make sure this is working as expected.

Your script does not seem to consider that some mp3s might already be there.  If your computer crashed in the middle of conversion you would be stuck redoing work already completed.

apt minimization

While it is a tiny part of what you're doing the apt portion of things could be more efficient.  You could cache whether these commands are installed by touching a file like ~/.ready_to_convert and hoping they don't get uninstalled by accident.  You could also choose to skip the dpkg-query and just see if which ffmpeg gives you any output.  This would let it work even if the commands are installed manually in /usr/local/bin without a package.

It would also be nice to only run apt-get update once.  You could set a flag like so:

if ! dpkg-query -W -f='${Status}' ffmpeg | grep "ok installed"; then
    apt-get update  && apt-get install ffmpeg
    APT_UPDATED=1
fi

if ! dpkg-query -W -f='${Status}' parallel | grep "ok installed"; then
    if [[ $APT_UPDATED < 1 ]]; then
        apt-get update
    fi
    apt-get install parallel
fi

But presumably the conversion is consuming most of the time so this will only let that get started a second or two quicker.


Testing class that transforms JSON into a spreadsheet

Ideally,
the source data and the expected result of the function under test are both visible inside the test case.
That readers can see clearly what is being tested and what the function does.

When the source and result data are relatively big,
it's better to not include in the test case,
but to keep in files in test resources.
This is usually not great, as readers lose visibility of what's really being tested,
but if large data pollutes the test case making it unreadable,
that's a worse option.

In your case, only the source data (json) is in the test case,
the result (xlsx) is outside.
In this case,
since the JSON is pretty unreadable in the code,
I recommend to move it to a resource file, like you did with the xlsx.
I think that's the most maintainable option for you.


Makes sense, but you should DRY up your concrete factory at least. Forgive me my rusty Java, but something like this:

public class ProductDocumentGeneratorFactory implements
        DocumentGeneratorFactory {

    public Document createDocument(String scheme, String authority, String path)
            throws ParserConfigurationException, SAXException, IOException {

        Uri.Builder uri = createUri(scheme, authority, path);
        Document doc = createDocument(uri);
        return doc;
    }

    public Document createDocument(String scheme, String authority,
            String path, HashMap<String, String> parameters)
            throws ParserConfigurationException, SAXException, IOException {

        Uri.Builder uri = createUri(scheme, authority, path);
        appendParameters(uri, parameters);
        Document doc = createDocument(Uri.Builder uri);
        return doc;
    }

    private Uri createUri(string scheme, string authority, string path)
    {
        Uri.Builder uri = new Uri.Builder();
        uri.scheme(scheme);
        uri.authority(authority);
        uri.path(path);
        return uri;
    }

    private Document createDocument(Uri.Builder uri)
    {
        URL url = new URL(uri.toString());

        DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
        DocumentBuilder db = dbf.newDocumentBuilder();
        Document doc = db.parse(new InputSource(url.openStream()));
        doc.getDocumentElement().normalize();

        return doc
    }

    private void appendParameters(Uri.Builder uri, HashMap<string,string> parameters)
    {
        Set<Map.Entry<String, String>> set = parameters.entrySet();
        for (Map.Entry<String, String> params : set) {
            uri.appendQueryParameter(params.getKey(), params.getValue());
        }
    }
}

Of course you can expand on that by employing the template pattern for the extra parameter settings.

I would also look into extracting the parameters to a "parameter object" instead of several parameters.


You could already create the StringBuilder with the text you want to 'splice', rather than building it up again. Then you can just determine how many newline characters you want to add (and you should use Environment.NewLine instead of specifying it explicitly), and insert them in the right place.

    private string SpliceNoteText(string text, int maxWidth)
    {
        StringBuilder sb = new StringBuilder(text);

        for (int i = 0; i < (sb.Length / maxWidth); i++)
        {
            int insertPosition = i * maxWidth;
            sb.Insert(insertPosition, Environment.NewLine);
        }

        return sb.ToString();
    }


From a once over:

I like how you store both el and $el
I also like how you expose your defaults
Not sure about _t, I think generally self is considered to a better name for this
You should have a spot in your boilerplate to define the prototype of your plugin class
Lint finds nothing worth mentioning


Average of positive and negative part of numpy matrix

There are a few things which we could improve:

When importing a module, you would not usually import all symbols from that module. You can however rename the module to make it more convenient, e.g.:

import numpy as np
# now we can do np.matrix(…), np.transpose(…), etc.

Instead of initializing a matrix and then reshaping it, create it directly in the shape you need:

exposures = np.zeros((numPaths, numExposures))

When defining a function put the function body on a new line, no matter how short it might be:

def plusv(x):
    return max(x, 0)

Instead of stuffing this normalization into a custom function, you could just use the builtin clip. It takes two arguments: a minimum and maximum bound, both inclusive. So plusv(a) would be equivalent to a.clip(0, np.inf) for floats and a.clip(0, np.iinfo(np.int).max) for integers.

The numpy functions can either be called as functions or as methods. Using them as methods is usually easier to read for people used to reading left-to-right.

ee  = exposures.mean(axis=0).transpose()
epe = exposures.clip(0, np.inf).mean(axis=0).transpose()
ene = exposures.clip(-np.inf, 0).mean(axis=0).transpose()

Oh dear, we are repeating code! Well, let's just use a list comprehension/generator expression instead:

positive_exposures = exposures.clip(0, np.inf)
negative_exposures = exposures.clip(-np.inf, 0)
ee, epe, ene = (xs.mean(axis=0).transpose() for xs in
    (exposures, positive_exposures, negative_exposures))

In the above snippet, the temporary variables positive_exposure and negative_exposure are entirely optional, but they make the code more self-documenting.

Note that I use snake_case for my variables. This is the convention in Python for normal variables or method names. The lowerCamelCase scheme is not generally used. While I recommend you follow the general Python convention of words separated by underscores, the only important thing is to be consistent. Whatever your naming scheme, names like ee and epe are not acceptable. Prefer full words over abbreviations, unless the abbreviations are common enough (i for an index is ok, and everyone understands str means “string“).


Your swap_mem will get called O(n log n) times. This makes it worth taking a closer look at for optimization.

One thing you can do is avoid repeated calls to malloc and free and instead preallocate the tmp buffer in the root function call and pass it around or use alloca to allocate the buffer on the stack.

void swap_mem(void * ptr1, void * ptr2, size_t size){
    /*
     * Swaps the values at the addresses pointed at by ptr1 and ptr2
     */
    void * tmp = alloca(size); //will get freed automatically on function exit.
    memcpy(tmp, ptr1, size);
    memcpy(ptr1, ptr2, size);
    memcpy(ptr2, tmp, size);
}


Before I comment on your implementation. Your code has a bug in it.

The code inside this if block:

if(modalShowPassword || modalPassword) {

will execute if either modalShowPassword or modalPassword is set but you really only want it to execute if both are set. You want to use the and (&&) operator like this:

if(modalShowPassword && modalPassword) {

With your code, think about what would happen if you were to have more (say 10) password boxes on your page? Would you repeat your code 10 times with different variables and different values for the data-js attribute? What I would suggest is to use something similar to the Bootstrap style (see how their modal dialogs work here https://getbootstrap.com/docs/4.1/components/modal/#via-data-attributes). There are two things to note. First they use the same data attribute for all instances even if there are multiple modals on the page. Secondly they do not apply an attribute to the modal form. They have two attributes on the toggle button, data-toggle defines the action while data-target tells the button what to show as a modal.

So a solution might look like:

<form>
    <input id="password1" type="password" autocomplete="current-password" name="password" placeholder="Enter Your Password">
    Show Password

    <input id="password2" type="password" autocomplete="current-password" name="password" placeholder="Enter Another Password">
    Show
</form>


"The Love Letter Mystery" - storing and comparison of strings

Honestly, this seems pretty good. There might be a faster way to do it, but it's not coming to mind.

I'd maybe not declare your int i that early on, and use the standard format instead, as that's what's expected.

for(int i = 0; i < n; i++)

As well as this, I'd give n a more descriptive name, such as num_lines or something similar. The same can be said for the other variables. ops and pos could give more information about what they're doing.


left = 2 * i + 1; for a non-random access iterator this is an operation that is linear with regards to i so it's slow for std::list as well.

Heapify can be adust to not need to std::advance an arbitrary amount by going over the list multiple times:

template<typename BiDirIterator, typename Cmp>
void heapify(BiDirIterator begin, BiDirIterator end, Cmp cmp)
{
    bool changed = false;
    do{
        auto parent = begin;
        auto left = begin == end? end : begin+1;
        auto right = left == end? end : left+1;
        while (left != end) {
            auto largest = left
            if (right != end && cmp(*left, *right))
                largest = right;

            if (cmp(*parent, *largest)){
                std::swap(*parent, *largest);
                changed = true;
            }

            parent = ++parent;
            left = right == end? end : right+1;
            right = left == end? end : left+1;
        }
    }while(changed);
}

This will loop over the data at most O(log n) times and puts the elements in heap order.

However for pulling out the data there is no way to only look at the affected elements in O(log n) time. Which means that you can't sort a linked list with heapsort and still be in O(n log n) time complexity.

Instead to sort a linked list in O(n log n) time you would use merge sort. This requires that you are able to be able to split the list up into separate linked lists in that you can merge. std::list has this functionality with it's splice member function. (It also has a sort member function that will do that built-in)


Shortcut script for elusive grep command

I'd lean towards find.  I'll provide some code review type comments at the bottom, but first a rewrite:

search() {
    local extensions
    local pattern
    local find_cmd
    local OPTIND opt

    local usage=$( cat - << END
Usage: $FUNCNAME [OPTION] ... PATTERN [FILE] ...
Search for PATTERN in each FILE.
Example: $FUNCNAME -t c -t h 'hello world' /code/internal/dev/ /code/public/dev/

Output control:
  -t    limit results to files of type
END
)
    if [[ $1 == --help ]]; then
        echo "$usage"
        return
    fi

    extensions=()
    while getopts ":ht:" opt; do
        case $opt in
            h) echo "$usage"; return;;
            t) extensions+=("$OPTARG");;
            ?) echo "invalid option: -$OPTARG";;
        esac
    done
    shift $((OPTIND-1))

    if (( $# == 0 )); then
        echo "no search term provided"
        return
    fi

    pattern=$1
    shift

    if (( $# == 0 )); then
        echo "no directories provided"
        return
    fi
    # your directories to search are now the positional parameters

    find_cmd=(find "$@" '(')
    or=""
    for type in "${extensions[@]}"; do
        find_cmd+=($or -name "*.$type")
        or="-o"
    done
    find_cmd+=(')' -exec grep -I "$pattern" '{}' +)

    "${find_cmd[@]}"
}

Notes

[ $1 = --help ] will give you syntax error if $1 is blank -- you get [  = --help ] and the = operator requires 2 operands. bash's [[ ... ]] is smarter about not dropping operands just because they're empty
You can build up arrays bit-by-bit with arr+=("$element")
you want to use more double quotes for your variables. If the pattern is "hello world", then grep $pattern file1 will look for the pattern "hello" in files "world" and "file1" -- grep "$pattern" file1 will work as you expect.
I use (( ... )) arithmetic expression for numeric comparisons.
I build up the find command piece by piece and execute it with "${arr[@]}" -- that specific syntax, indexing the array with [@] and surrounding it with double quotes, is the way you'll want to expand arrays most of the time. That specific syntax will expand the array into elements, but it will keep elements containing whitespace as a single argument.

This includes using "$@" over $@ -- the former gives you the actual arguments given by the user, the latter gives you all the words in the arguments. A demonstration:

set -- "foo bar" "hello world"    # set the positional parameters
printf "%s\n" "$@" | wc -l        # print the parameters one per line
                                  # and count the lines -- 2
printf "%s\n" $@ | wc -l          # here you get 4

if t is an array, $t gives you the first element only.


It's more idiomatic to use .map on the stream instead of Collectors.mapping:

stringsMaybe.stream()
    .filter(Optional::isPresent)
    .map(Optional::get)
    .collect(toList());

Without introducing a helper method or a custom collector, that's the shortest and clearest way to do this.

Since Java 9, Optional offers a stream method, enabling you to do .flatMap(Optional::stream) instead of .filter(...).map(...).


Implementing Linked lists in Python from scratch

Review

You should take advantage of the bi-directional nature of a doubly linked list. It's a pitty to let it use navigation of a normal linked list just for the sake of enabling inheritance. And why should a normal linked list be able to branch between normal and doubly mode? This is a code smell.

A practical way to create a doubly linked list, is to create a circular list. You only need to store the head. tail would be head.prev. This also works with a single node head = head.prev = head.next. The advantage is that less if statements are required to perform insert / delete operations. Walking the nodes starts at head and stops until we encounter head again. We could walk both directions if we wish to.

You can initialise a single node:

self.head = ListNode(x, doubly)
self.head.prev = self.head
self.head.next = self.head

The delete would be significantly simplified:

After walking the nodes until you get the proper node p given x, you can call

if p.next == p:
    self.head = None
else:
    p.next.prev = p.prev
    p.prev.next = p.next
    if self.head == p:
        self.head = p.next
del p
self._length -= 1

I don't see a value in using inheritance for these lists.

I also don't get why you store p = self.head in insert. It's an unused variable.


Calculating a pattern-match score on a ten-element array

This whole approach is completely bogus and not worth your time.

A "pattern" is simply an array of percentage changes. For simplicity let's use a pattern size of 5. So a pattern might look like:

pat1 = [ 1%, 3%, -2%, 1.6%, 2.1% ]

meaning that the price difference (from the price at time 0) at time 1 was 1%, at time 2 was 3%, at time 3 was -2%, etc.

Given another pattern we want to compute how "similar" they are. This is done componentwise using a similarity function for percentages whose definition is:

sim(p,q) = 100 - abs (percentageChange(p,q))

where percentageChange is the percent change between p and q with an absolute value in the denominator:

percentChange(p,q) = 100 * (p-q)/abs(q)

So in python, sim is:

def sim(p,q):
  """Compute the similarity between two percentage changes."""
  pc = 100 * (p-q) / abs(q)
  return 100 - abs(pc)

Note since we are multiplying everything by 100 that the 1% value in pat1 is just the value 1, not the conventional 0.01.

Given two patterns, pat1 and pat2, we say they are close if

sim( pat1[i], pat2[i] ) >= 70   for each i in 0..4

The big idea

The idea behind these patterns and similarity measures is that given historical data we compute a pattern and a future price change with the idea that the pattern implies the future price change.

Then given current data we compute the current pattern and search for a similar pattern in our database from which we can deduce the corresponding price change.

The main problem is that the sim function is not symmetrical in its arguments, i.e. sim(p,q) and sim(q,p) may have wildly different values.
Consider these examples:

sim(1.200, 2.000)   = 60.000000
sim(2.000, 1.200)   = 33.333333

sim(-1.000, 2.000)  = -50.000000
sim(2.000, -1.000)  = -200.000000

sim(-1.000, -2.000) = 50.000000
sim(-2.000, -1.000) = 0.000000

sim(-1.000, 1.000)  = -100.000000  # guess these are actually the same!
sim(1.000, -1.000)  = -100.000000

sim(-1.000, -0.100) = -800.000000
sim(-0.100, -1.000) = 10.000000

sim(-1.000, 0.100) = -1000.000000
sim(0.100, -1.000) = -10.000000

A better definition for sim(p,q) would simply be:

sim(p,q) = abs(p-q)

and then to define two patterns to be close if

sum of sim(pat1[i], pat2[i]) for i in 0..4
  is LESS THAN a certain number

The prediction algorithm then boils down to a nearest neighbor search in a high number of dimensions (e.g. the length of the pattern array.)


Substring a String from given start index and given substring length

for something like this I would suggest making it an Extension Method.

To that end I would suggest a name change as well.  Perhaps Substring_NoException.

To make it complete I think having the same overloads as Substring would help.

As for reducing/eliminating the conditional checks, one option is to use a Try/Catch block and simply catch every exception that Substring would and return an empty string except for the length being too long.

It might look something like this:

Imports System.Runtime.CompilerServices

    ''' <summary>
    ''' Extension method for <c>Substring</c> to return empty strings instead of throwing exceptions
    ''' </summary>
    ''' <param name="startIndex">The zero-based character position of a substring in this instance</param>
    '''
    <Extension()>
    Public Function Substring_NoException(input As String, startIndex As Integer) As String
        Return input.Substring_NoException(startIndex, input.Length - startIndex)
    End Function

    ''' <summary>
    ''' Extension method for <c>Substring</c> to return empty strings instead of throwing exceptions
    ''' </summary>
    ''' <param name="startIndex">The zero-based character position of a substring in this instance</param>
    ''' <param name="length">The number of characters in a substring</param>
    '''
    <Extension()>
    Public Function Substring_NoException(input As String, startIndex As Integer, length As Integer) As String
        Try
            Return input.Substring(startIndex, length)
        Catch ex As ArgumentOutOfRangeException
            If ex.ParamName = "length" AndAlso length > input.Length - startIndex Then
                Return input.Substring(startIndex)
            End If
            Return String.Empty
        End Try
    End Function

usage is:

Dim testString As String = "abcdef"
Dim testString2 = testString.Substring_NoException(3, 100)

Returned value is:

testString2 = "def"


Data Structures BFS Breadth First Search algorithm improvement written in python

The very first thing that caught my eye was the del queue[0] which is a \$O(n)\$ operation since it requires for the whole list to be moved.

There is a quick win here - use collections.deque() which has a .popleft() method operating at \$O(1)\$:

queue = deque([node])

while queue:
    node = queue.popleft()

    for i in self.graph[node]:
        if visited[i] is None:
            queue.append(i)

            visited[i] = True
            distance[i] = distance[node] + 1
            parent[i] = node

where deque is imported this way:

from collections import deque

Also note the use of while queue: instead of a less efficient and less readable while len(queue) != 0:.

There is a number of code style issues as well:

look for the PEP8 violations, in particular:

class name should start with a capital letter - Graph instead of graph

use proper documentation strings instead of regular comments before the method definitions

watch for the spaces around operators and the use of blank lines

_ (underscore) variable name is agreed to be used a throwaway variable name, e.g.:

self.graph = [[] for _ in range(n)]

instead of:

self.graph = [[] for y in range(n)]


1.This handler isn't asynchronous.

The easiest way to accomplish this with tornado is to make a coroutine like so:

@gen.coroutine
def get(self, word):
    driver = webdriver.PhantomJS(executable_path=r'bin/phantomjs')
    yield driver.get(url="http://dictionary.cambridge.org/dictionary/english/%s" % word)
    is_word_cached = self.check_cache(word)
    if is_word_cached:
        response = {'level': is_word_cached[0][0]}
    elif self.check_word_404(driver):
        response = {'level': "This word wasn't found"}
    else:
        try:
            level = driver.find_element_by_xpath(xpath="//span[@class='def-info']/span[contains(@title,'A1-C2')]")
            level = level.text
        except NoSuchElementException:
            level = "The word level isn't known"
        self.write_cache(word, level)
        response = {'level': level}

    self.write(response)

The main thing is to decorate the get method with @gen.coroutine and then have a yield statement in the blocking line, in your case the fetching of the data. Documentation here

You can store the cache in a in-memory database like Redis, Memcached, in plain Python or in a pickle.


Determining if an std::string contains a numerical type

Functional issues

Your code doesn't consider negative numbers to be integral. You're also missing support for things like 1e4 for floating point. Ultimately, std::stoi and std::stof might be better for your needs:

boost::optional<int> convert_integer(std::string const& str) {
    try {
        size_t pos;
        int result = std::stoi(str, &pos);
        if (pos == str.size()) {
            return result;
        }
        else {
            return boost::none;
        }
    }
    catch (std::invalid_argument& ) {
        return boost::none;
    }
    catch (std::out_of_range& ) {
        return boost::none;
    }
}

and similar for floating point.

is_digit

There's no reason for this to be a lambda. It's actually quite a bit shorter if you just write it as a function:

const auto is_digit = [] (const char c) { return std::isdigit(c); };
bool is_digit(char c) { return std::isdigit(c); }

using std::cbegin;

There's actually no reason to bring this in, since you could already make an unqualified call to cbegin() thanks to ADL. Also, you don't even need cbegin() since begin() would have done the same thing (str is a reference to const in all of your functions).

Too much action

This is really busy:

return it != cend(str) && *it++ == '.' && std::all_of(it, cend(str), detail::is_digit);

And it's confusing since you increment it in the middle. Which happens to work, but I had to stop and really think about it to convince myself that it does. You could simply shift the ++ into the all_of:

return it != cend(str) &&
    *it == '.' &&
    std::all_of(it + 1, cend(str), detail::is_digit);

Since there's no modification going on here, each of the 3 parts are easier to reason about.

Just std::string?

A logical extension would be to change your functions to take iterator pairs, or const char* pairs. You could provide convenience string methods to forward to the helpers, but it'd be potentially more useful to have

// one of...
bool is_integral(const char* first, const char* last);
template <class It>
bool is_integral(It first, It last);

// ... and a helper to forward
bool is_integral(const std::string& str);


strings are probably the wrong choice to represent roles. You lose the power of the type system -- the compiler won't pick up on typos, e.g. "Amdin". Consider using an enum instead.

Be consistent with your naming. Why is it USER_ROLES.IsRead and USER_ROLES.Write (not IsWrite)? Stick to the standard naming conventions, e.g. write UserRole.IsRead instead of USER_ROLES.IsRead.

Does the calling code really need an array returned? It would probably be fine with an IEnumerable.

If we change the return type to IEnumerable<UserRole>, we can write

public IEnumerable<UserRole> GetRoles()
{
    if (this.IsAdmin)
    {
        yield return UserRole.Admin;
    }

    if (this.IsRead)
    {
        yield return UserRole.Read;
    }

    if (this.IsWrite)
    {
        yield return UserRole.Write;
    }

    if (this.IsEntityUser)
    {
        yield return UserRole.EntityUser;
    }
}

But I'm guessing you're asking more about the repetition. Here's one way to get around it -- create an association between the booleans and the values. Here I've used a dictionary; it's possibly a bit of overkill but it does give us the nice initialiser syntax:

public IEnumerable<UserRole> GetRoles()
{
    var roles = new Dictionary<UserRole, bool>
    {
        { UserRole.Admin, this.IsAdmin },
        { UserRole.Read, this.IsRead },
        { UserRole.Write, this.IsWrite },
        { UserRole.EntityUser, this.IsEntityUser }
    };
    return roles.Where(role => role.Value)
                .Select(role => role.Key);
}

Other options would be an array of KeyValuePairs, an array of Tuples, etc.


Correct 4 bytes out of 65K to match a crc32 checksum

Disclaimer: It is not a review, and I am wild guessing.

The DrDobbs article outlines how the partial CRC could be recombined. I recommend to precompute CRCs of all fixed parts of the chunk, and let the GPU handle the rest. GPUs are notoriously good in number crunching, and you have at least 16K+ truly parallel cores on your graphics card. It is \$2^{18}\$ search space per core. Seems feasible.


Your test code doesn't match your test output. "Test.cpp" isn't even valid C++ code!
How does your massive amount of code do anything significantly different from the following eight lines?
template<class T>
struct Singleton {
    struct Token {};
    static T& getInstance() {
        static T instance(Token{});
        return instance;
    }
};

Example usage:
struct Widget { explicit Widget(Singleton<Widget>::Token); };
Widget& example = Singleton<Widget>::getInstance();

As for why the "thread-safe statics" feature doesn't work on your freestanding platform, I would suspect it's because your freestanding platform doesn't support thread-safe statics. :)  Check the documentation for your toolchain to see how static variables are handled on that particular platform.


My understanding:

The first if statement is meant to check whether a thread is already executing the loop by checking whether cancellationTokenSource is null; if it's not null, then signal that a sync is required by setting syncRequired to true and returning. The thread executing the loop will then run another loop due to the while (dataWasSynchronised || syncRequired ) condition.

If cancellationTokenSource is null, then the sync loop isn't being executed, so this thread goes on to do just that.

private async void StartSynchronisation()
{
    // the cancellation token operates as the flag to indicate if the operation is running
    if (this.cancellationTokenSource != null)
    {
        syncRequired = true;
        return;
    }

    try
    {
        this.cancellationTokenSource = new System.Threading.CancellationTokenSource();

So the first problem is that that check (the first if statement) isn't synchronised. One thread can see cancellationTokenSource is null and move on to execute the sync loop, first setting cancellationTokenSource to a new object. Before it assigns the new object - say, after setting syncRequired to true - there's a context switch, and another thread executes the same code. It too sees that cancellationTokenSource is null, sets syncRequired to true, assigns a new object to cancellationTokenSource, and starts running the loop. Then there's another context switch and our original thread is back again, assigns a new object to cancellationTokenSource, and starts executing the loop as well. Voila, you've now violated your first requirement of having zero concurrency.

The fix is simple: the check, as well as any objects being altered to affect the outcome of the check, must be synchronised.

A basic way to achieve this:

class MyClass
{
    private readonly object syncLock = new object();
    private volatile bool syncRequired = false;
    private volatile bool syncRunning = false;

    private void MySyncFunc()
    {
        lock (syncLock)
        {
            if (syncRunning)
            {
                syncRequired = true;
                return;
            }

            syncRunning = true;
        }

        while (true)
        {
            syncRequired = false;

            // your sync operations here

            lock (syncLock)
            {
                if (!syncRequired)
                {
                    syncRunning = false;
                    break;
                }
            }
        }
    }
}

I'll quickly point out this isn't the most efficient or pretty, but it's a simple (and hopefully accurate) way to show and do it.

First of all notice you can't even check whether your sync loop is running without getting the lock. So only one thread at a time can check. If a thread determines it isn't running, it sets the value to true (to signal that it's now running) and exits the lock. The second thread that arrived on the scene in the earlier example, if it was context-switched to during that time, would not have been able to proceed because it couldn't get the lock. Once the first thread exits the lock, the second thread gets the lock. Now it sees that syncRunning is true, so it sets syncRequired to true and returns. No sync problems there.

Now let's look at the loop. Nevermind that it's an infinite loop, I just wanted to get the checks out of the loop condition. So first thing it does is set syncRequired to false. This doesn't need to be synchronised because even if another thread had set it to true right before, you're gonna go and do the actual synchronisation right now anyway.

So after your own sync stuff completes, it's time to check whether you can exit the loop or need to repeat it. (I've left out checking the result of your sync operation as it's not relevant to this.) Again you got to get the lock. Why? Because the thread could check syncRequired and see it's false, and move on to set syncRunning = false. But before it's set, context switch, another thread checks and sees that syncRunning == true, so sets syncRequired to true and returns. Meanwhile the first thread continues to set syncRunning to false and returns as well. Now you have syncRequired == true but no thread running the loop. Whoops, you missed an event! Take a look at your original code and you'll see this problem is there, if syncRequired is set to true after the loop exits and before cancellationTokenSource is set to null in the finally block.

So what the code above does instead is it gets a lock on the same object again and checks whether a sync is required. If it is, fine, the lock is exited and the loop starts over. But if it's not, then it sets syncRunning to false before it exits the lock. Since any other thread that wants to check if syncRunning == true has to wait until the lock in the loop is exited, they will see the correct value when they finally do get the lock, because if the first thread will exit the loop, it will set syncRunning to false while it has the lock and before it exits the lock. Whether the next thread gets the lock and checks, it will see that syncRunning == false, and go on to run the loop itself. No more lost events!

Couple other issues:

If you're going to use a member variable in a loop, but not modify its value in the loop (or only ever set it to a single value on each iteration such that there is no apparent point in doing so, even though you know that some other thread can modify the value), then mark it volatile. This will result in the value being read from memory each time, so you pick up changes made from other threads. Otherwise the compiler could optimise the read out of the loop; after all, it's never changing during the lifetime of the loop.

And your code isn't cancellable. Sure you create a CancellationTokenSource, but you never actually use it - hence why I simply substituted it with a bool in my example. What you need to be doing is checking the token each iteration (or possibly pass it to the function being called if that's the part that should allow cancellation of its task, though you'd then have to alter the function to accept and know how to use the token) and seeing whether a cancellation has been requested; then if so, throw a OperationCanceledException (or however you want to abort what's currently running).


It may be simpler (and definietly faster when it comes to larger data) to create a private static final Map<Character, Integer> with LETTER -> VALUE mapping prior to any calculations. Besides repeating the put many times when constructing it, you can be even cooler although less readable IMO:

private static final Map<Character, Integer> LETTER_TO_POINT;

static {
    final Map<Character, Integer> letterToPoint = new HashMap<>();
    // letterToPoint.put('A', 1);
    // letterToPoint.put('B', 3);
    // booooooring :-(

    BiConsumer<String, Integer> rememberPointValues = (letters, points) -> {
        for (char letter : letters.toCharArray()) {
            letterToPoint.put(letter, points);
        }
    };
    rememberPointValues.accept("AEILNORSTU", 1);
    rememberPointValues.accept("DG", 2);
    rememberPointValues.accept("BCMP", 3);
    rememberPointValues.accept("FHVWY", 4);
    rememberPointValues.accept("K", 5);
    rememberPointValues.accept("JX", 8);
    rememberPointValues.accept("QZ", 10);
    LETTER_TO_POINT = Collections.unmodifiableMap(letterToPoint);
}

Once you have that, the charPoint method is much simpler:

private int charPoint(char ch) {
  return LETTER_TO_POINT.get(ch);
}

Moreover, you can iterate over the letters in the string like that:

for (char letter : word.toCharArray()) {
  score += charPoint(letter);
}


I would advise you not implementing stacks with a linked list because for each char of info you consume 2 * sizeof(void*) because of memory alignment in your struct.

Because of this you won't be able to handle a big amount of data.

What happens if you push EMPTY_STACK ? Your pop will return a false EMPTY_STACK when going through the if. Isn't there a better way to handle this ?


Take actions depending on whether a number is within X percent of another

Apart from replacing the second if with an else, I would define this as a separate function:

function withinTolerance($value, $target, $tolerance) {
    return $value >= $target * (1.0 - $tolerance) && $value <= $target * (1.0 + $tolerance);
}

Then your call can just be:

if (withinTolerance($x, 100, 0.05)) {
    // Within tolerance
} else {
    // Outside tolerance
}

I made everything one line as it's a simple expression, which should clearly indicate what it does, but you can feel free to break $target * (1.0 - $tolerance) to $min, and $target * (1.0 + $tolerance) to $max if you like.

This allows you to change the target and tolerance easily, and also has the added bonus of being reusable, and testable. (How do you go about verifying that your current version works, for example?)

You can define some simple tests:

if (withinTolerance(99, 100, 0.05)) { echo "Pass"; } else { echo "Fail"; }
if (!withinTolerance(10, 100, 0.05)) { echo "Pass"; } else { echo "Fail"; }

And so on.

You could also use a trick with the absolute value (as mentioned in comments on the question), but that takes a lot of the meaning away. This version allows you to easily define upper and lower bounds:

function withinSingleTolerance($value, $target, $tolerance) {
    return withinTolerance($value, $target, $tolerance, $tolerance);
}
function withinTolerance($value, $target, $toleranceBelow, $toleranceAbove) {
    return $value >= $target * (1.0 - $toleranceBelow)
           && $value <= $target * (1.0 + $toleranceAbove);
}


Convert human readable string to TimeSpan

public static class TimeSpanConverter
    {
        public static TimeSpan Convert(string input)

To people who use WPF a lot, the name hints at this class being a TypeConverter or an IValueConverter. I would say that it's really a parser. Similarly, I think that the method should be called Parse.

            var units = new Dictionary<string, int>()
            {
                {@"(\d+)(ms|mili[|s]|milisecon[|s])", 1 },
                {@"(\d+)(s|sec|second[|s])", 1000 },
                {@"(\d+)(m|min[|s])", 60000 },
                {@"(\d+)(h|hour[|s])", 3600000 },
                {@"(\d+)(d|day[|s])", 86400000 },
                {@"(\d+)(w|week[|s])", 604800000 },
            };

The prefix milli has two ls. millisecond ends in a d.

[|s] is more idiomatically written as s?.

Given the way the regexes are used (searching for matches without anchoring), most of them could be simplified. E.g. any match for min would also match m, so @"(\d+)m" will find exactly the same multiples of a minute.

The numbers on the right are magic numbers, and at a glance I can't be completely certain that they're correct. I would prefer to use TimeSpan instances and take advantage of TimeSpan.FromMilliseconds(1), TimeSpan.FromSeconds(1), etc.

I think that a real world example of a human-readable string would be "1 day, 13 hours, 2 minutes, 12 seconds, and 52 ms". At the very least, I would add \s* between the digits and the units.

Another quite realistic example would be "2 minutes and 13.5 seconds".

It's also plausible that people will use : as a separator: "2 days, 11:04:20"

Then you get ugly stuff like months and years, which don't have fixed lengths.

If you want to refuse to parse some of these, that's fine. But in that case, I think the method still needs to handle them by throwing an exception. At present they would all return TimeSpan.Zero, which is misleading.

Have you given any thought to localisation? If you only want to use this in an English-language context, wouldn't [0-9] make more sense than \d? If you're matching all digits, you should have test cases for things like "¹day١hourminute①second".

            var timespan = new TimeSpan();

Can you think of a more descriptive name for this variable?

                var matches = Regex.Matches(input, x.Key);
                foreach(Match match in matches)

If the string has more than one match for a given regex, isn't that rather worrying? I wouldn't consider "4hours2minutes1hour" to be a very realistic input.

                    var amount = System.Convert.ToInt32(match.Groups[1].Value);
                    timespan = timespan.Add(TimeSpan.FromMilliseconds(x.Value * amount));

Again, amount isn't a very descriptive name.

TimeSpan supports the operator +, and IMO total += addend; is more readable than total = total.Add(addend);.


MongoDB - Find records in one collection that match string in 2nd collection

You're turning the result of db.codes.find() into a list, but you're just iterating over it anyway. Why not just leave it in whatever form it arrives? If it can be converted to a list it can probably be iterated over. This would save you the time of conversion and mean you might not need to have a full list stored in memory.

Also x isn't a great name for the temporary value when you could just use code and take up only 3 more letters.