Built to Scale: Running Highly-Concurrent ETL with Apache Airflow (part 1)

Apache Airflow is a highly capable, DAG-based scheduling tool capable of some pretty amazing things. Like any other complex system, it should be set up with care. The following is an overview of my thought process when attempting to minimize development and deployment friction.

Apache Airflow has seemingly taken the data engineering world by storm. It was originally created and maintained by Airbnb, and has been part of the Apache Foundation for several years now. After heavily leveraging it for about a year (almost 2 million ¡idempotent! ETL tasks later) and seeing its full potential (but numerous drawbacks), I was tasked with streamlining the deployment and operation of the system.

The obvious first step? Sit down and note everything I loved about Airflow…. and everything I hated.

The list went something like this.

Apache Airflow ❤’s

Code-defined DAGs

It is incredibly easy to define, manage, and communicate complex ETL dependencies to stakeholders.

The UI and corresponding operational functionality

The breakdown of dags, dag runs, tasks, task log output, etc, is very nice. The UI also includes features like Gantt charts, task duration visualizations, immediately-visible DAG definitions, and more.

System composition

As an engineer, the associated subsystems involved with running an Airflow instance are very clean. The system consists of a scheduler process, a worker process, a webserver process, a message broker, and a database backend. IE: it’s a real application.

Retry functionality

Since stuff happens and systems go down, ETL should almost always be wrapped in retry functionality. Airflow makes this simple, as its BaseOperator class (from which operators inherit) includes the ability to specify various retry strategies. Want to simply retry a few times on a specified delay? You can. Want to declare an exponential backoff? You can do that too.

ETL backoffs can be cleanly implemented with retrying and/or tenacity, but Airflow includes this functionality out of the box.

Airflow Contexts

template_context is a relatively-undocumented piece of Airflow magic, and provides rich metadata at the task instance level. It is highly useful for stamping load metadata into the contents of the data itself, or notifying when a particular job fails or goes into a retry loop.

Whatever you do, set provide_context: True when defining your DAGs.

“It’s just Python”

Airflow is a (very) complex system, and the fact that “it’s just Python” speeds up the process of coming up to speed with how the thing works. Digging around in the source code is often necessary, and made much easier by the fact that it’s written in a readable language.

Apache Airflow 💔’s

Limited deployment documentation

I say limited, but I mean very little. You will need to dig around to find systemd documentation and a few unit files, but these will need to be modified according to your host os.

Limited community information on good deployment strategies

After running Airflow for months, scheduler and worker downtime on each code deploy was something I came to consider unacceptable - especially as our data engineering team grew. Airflow was getting in the way of getting code out; every deployment meant that all dags had to be manually disabled, running task instances bled off, the system shut down, new code pushed, system turned back on, and then all dags re-activated… by hand.

Most of the “production” Airflow deployment articles I read included dropping the system into the background with nohup, docker-compose, or even running in a screen session. While these are interesting strategies, I wanted something I could run as a real service and systemctl or journalctl as necessary. I wanted Airflow to run as natively as possible, with real logging, real Restart=always, and able to be systemctl enabled.

The tightly-bound nature of “plugins” code

This one still puzzles me. In most cases, Airflow does a wonderful job loosely binding components of its architecture. The system’s DAG-discovery capability is lovely. Copy a dag to the dag_folder of all nodes, and the system will pick it up according to the configured dag_dir_list_interval.

In other cases, it is very tightly bound. Just because the scheduling system is written in Python does not mean that your Python-based ETL code should be interwoven throughout global scope.

Does your JenkinsCI implementation (written in Java) depend on your Java application, in order to build and deploy said Java application? Do you need to redeploy Jenkins, whenever new code is pushed to your Java app?

I hope not.

Yet… this exact situation is all-too-easy to accomplish with Airflow.

Significant overhead when running multiple nodes

Many Airflow implementation posts I read only discussed running Airflow on a single node, with all components running in separate containers or separate nohup’d processes. This is great, but the system documentation indicates it is capable of much, much more. A LocalExecutor will work for a while, until you have hundreds of ETL tasks that must be run and stakeholders complaining that your system is down. Secondly, running all ETL on a single node is asking for trouble as it is a glaringly-obvious single point of failure.

Running jobs via a CeleryExecutor is the built-in way forward here, but several worker nodes must then be provisioned, identically-configured, and kept up to date on each code deploy. And this does not take into consideration the overhead of managing a (redundant, fault-tolerant) message broker.

It’s a lot to keep track of!

ETL is bound to blocks of time

If you incorporate the various ts, execution_date, latest_date context values into your ETL code, I will guarantee that it bites you when processing data from different timezones or regions. Airflow timestamps are relative to the host, while your Salesforce instance could be configured to America/Los_Angeles and your mysql database with (tz-naive) datetime values could be configured to America/New_York. I repeat: this will bite you.

“It’s just Python”

See The tightly-bound nature of “plugins” code above. The system being written in Python makes it very flexible. Often too flexible.

After thinking long and hard about how to make the system friction-free…

I came to the following requirements of a system refactor.

1. Infrastructure should be separated into functional components.

To minimize points of failure, Redis should not be running on the same instance as the scheduler process, the scheduler process should not be running on the same host as the worker process, etc. This infrastructure was to be deployed via AWS so I settled on ElastiCache Redis as message broker, RDS Mysql for the database backend, a (relatively small) EC2 instance for the webserver and scheduler, and separate (relatively large) EC2 instances for all worker processes.

2. Infrastructure should be replicable across environments and each environment should be easy to scale.

In order to stamp out infrastructure and provision Airflow nodes, Terraform with workspaces and remote S3 state was heavily leaned upon.

This makes it easy to terraform workspace select and terraform apply for setting up alternative environments. It also makes it easy to change a variable corresponding to the number of airflow worker nodes from 2 to 3 and spin up/provision another worker node.

3. Airflow should run as a native service on the respective host machine.

Instead of nohup-ing individual processes or managing a docker-compose-based implementation, each Airflow process (scheduler, webserver, flower, worker) was set up with its own unit file, systemctl enable’d (so it comes back when the machine restarts), and systemctl started.

Task log output is configured via the base_log_folder configuration variable and handled accordingly.

4. Airflow jobs should be executed across a number of workers

Apache Airflow ships with the ability to run a CeleryExecutor, even though it is not commonly discussed. It is the critical piece to distributing ETL tasks across a pool of workers. Provisioning and managing a broker adds overhead to the system, but is well worth the effort.

5. Airflow should not need to be restarted whenever new ETL code is deployed.

This is logically intuitive but relatively difficult to accomplish with Airflow. I chose to maintain clear separation between Airflow and task execution by running all tasks within docker containers. Airflow itself is set up as bare-bones as possible, and all task instances simply docker run img_name python3 some_entrypoint.py.

6. Fully-automated rolling deployments were necessary…

without toggling a single switch or waiting for a single DAG to complete.

One of the biggest barriers to deploying new changes was the fact that we had to pause all DAGs, wait until all tasks were complete, and redeploy the system each time code was modified. This amounted to over 30 minutes for each deploy, for at least two engineers, and introduced large amounts of toil and human error.

The solution has worked very well and looks like:

  1. Any time code hits the master branch of our core ELT/ETL code, the repo is packaged into an analytics Docker container, which is then deployed to all Airflow worker nodes.
  2. A dags directory is scp’d to the respective dag_folder on all Airflow nodes. These changes are picked up by each node and represented via the UI.
  3. All dags use a BashOperator, which docker runs the most recent analytics container and calls the appropriate code entrypoint.

Because all ETL runs via docker containers, preexisting processes do not need to be killed or monitored until completion. The changes are simply executed next time the process runs.

7. Insight into system state was necessary.

This primarily includes host-level metrics and system performance metrics, but is being continually built upon.

First and foremost, all machines are provisioned with node exporter by default and open up port 9100 as a scrape endpoint.

Secondly, (batch-based) ETL code leverages a push gateway for all metrics as these jobs are short-lived ephemeral processes.

Third, Flower runs as a service on the airflow scheduler node, so current Celery status is easily represented.

Performance so far

Like any initiative of reasonable size, this system refactor was measured by a number of critical metrics. I’ve used the following to represent my success in increasing reliability, increasing scalability, and decreasing deployment overhead.

319,390 Airflow task executions have completed successfully after refactor, across a number of workers.

The Celery failure rate is 0.0005%.

1 line of code must be changed to add 1-100+ additional Airflow worker nodes.

0 deployments have required human interference, and 0 human errors have been introduced.

Deploying to 2x the infrastructure is 40x faster than it was previously (with fewer machines).

In conclusion

I’ve seen very good success when automating the deployment of multi-node Apache Airflow infrastructure, and the above guidelines were critical to the success of this effort. The next post will get into the setup details, so please stay tuned.

Read more posts by this author.

Boston, Ma


dietitian newport beach

10 Golden Rules to the Healthiest You

10 Golden Rules to the Healthiest You

Follow these 10 Golden Rules to optimize your health with simple, everyday habits.

Increase Gut Bacteria, Registered Dietitian in Orange County

Truth: you can have low levels of gut bacteria even if you've been taking probiotics for years.

We live in a world where digestive issues are becoming more problematic and also, the new normal. As someone who has been working with clients who have a multitude of digestive issues  — bloating, poor absorption of nutrients, food intolerances, H. pylori, the list goes on — I know a thing or two about digestion both personally and professionally. Poor digestion can be caused by many factors that affect gut bacteria, including a poor diet (i.e. lack of whole foods, fiber, nutrients), repeated and frequent use of antibiotics and some prescription medications, stress, genetics, and underlying medical issues.

Until recently, scientists believed that taking probiotics was the best way to increase the levels of healthy gut bacteria. Yes, it is a magical pill filled with little bacteria that has the ability to regulate your gut health, however...

recent research has shown that while probiotics have numerous benefits, they don’t quantitatively increase beneficial gut microbes over the long term.

If not probiotics, then what does?

We have enzymes in our gut to break down large molecules, such as protein, carbohydrates and fat. That does not happen with fiber. Instead, it passes through our stomach, drawing out waste, and remains completely intact before entering the colon, where it becomes “food” for the beneficial microorganisms that live there.

Studies have shown that fiber has the incredible ability to increase levels of beneficial bacteria in the gut. Fiber is a “prebiotic,” which is defined as a nutrient that is not digestible by humans but that increases the levels of beneficial gut bacteria.

So, if you have low levels of beneficial bacteria, don’t forget to include fiber/prebiotics in your strategy, along with your probiotics.

They support each other: the probiotics are the beneficial bacteria themselves, and the prebiotics are the food these bacteria need to thrive.

Diet for Candida And Leaky Gut Syndrome

Referenced From The Ultimate Candida Diet Program

Nutritionist in Orange County for Weight-Loss and Lifestyle Change

Is it just me or is the new health trend starting to look like we are micromanaging our lives?

We’re tracking our calories, tracking our steps, restricting our carbs and counting our nuts. What happened to intuitively eating and exercising for fun? Has it come to running around our kitchen table just to fit in those 10,000 steps by the end of the day?

A study published in 2016 shows no difference whatsoever between those who used a fitbit for a year and those who didn’t. This kind of data should have you rethinking these expensive devices. Stop stressing about the numbers, for all we know it’s probably negating the health benefits of exercise!

I have even seen some patients who sacrifice their yoga workout to go on a run instead. That way, their  doing something that will add more steps to their day. What good is that? Yoga has many health benefits but the obsession with tracking steps seems to be able to override that.

The trackers are definitely pretty but the symbolism is not.

Trackers show the world we need to be monitored, and that is certainly the opposite of nutrition counseling (by a dietitian, at least.) As a Registered Dietitian, my job is to guide and educate my patients until they reach the point where they finally believe that they can trust themselves to make good food decisions.

Looking for some guidance on how to become an Intuitive Eater and ditch the tracking?


Loading Posts...

Follow Piarevaracien's Eternal Soldier through Belarus's Blood-Soaked Past on "U pošukach pačatku tych šlachoŭ"

Piarevaracien U Posukach Pacatku Tych Slachou Main

U pošukach pačatku tych šlachoŭ

Piarevaracien already delivered one of the musical highlights of 2016 with their album Spadčyna, which brought together traditional and modern interpretations of folk. Scarcely three months later, the Belarusians are already back, this time with a metal full-length titled U pošukach pačatku tych šlachoŭ.[1] This new release is, above all, a testament to the band’s versatility. With Spadčyna already standing as one of the year’s finest folk albums thus far, U pošukach pačatku tych šlachoŭ succeeds on its own terms, with its thirty-six-minute metal offering containing few, if any, boring or otherwise lackluster riffs. As a result, Piarevaracien’s new album is their second career highlight within a three-month period.

Piarevaracien’s two 2016 records are vastly different on the surface, with the respective timbre, instrumentation, and style of each album impossible to recognise as originating from the same collective. On a deeper level, however, there is continuity between the two releases. One shared trait is the consistent quality of both albums, which is remarkable in itself given how radically different they are musically. Yet, even more importantly, both Spadčyna and U pošukach pačatku tych šlachoŭ coil their respective themes into a cohesive narrative, which in turn helps each work shine as an artistic whole that rises far above the sum of its parts.

Where Spadčyna dealt primarily with the change of seasons narrated against a backdrop of more universal motifs, U pošukach pačatku tych šlachoŭ‘s thematic canvas is decorated only with war. The album’s central theme is ‘the eternal soldier’, who has risen up throughout history, destined to sacrifice his own life for his people and soil. The album’s seven tracks—titled simply ‘Razdziel’ I through VII—cover different periods of Belarus’s blood-soaked past.

‘Razdziel II’ reminisces about the Battle of Grunwald, which took place in 1410 and saw Poland and the Grand Duchy of Lithuania (which included the territory of present-day Belarus) decisively defeat the Knights of the Teutonic Order, halting their expansion Eastwards. This confrontation between East and West, which has been labelled a medieval rehearsal of the Battle of Stalingrad, is only one example of Belarus’s long-lasting affair with armed struggle; the album continues to tell of the Belarusian resistance against the Soviet Union, both in the wake of the Russian Revolution and during (and even after) World War II. Threading through these historic accounts is the eternal soldier, whose presence ties said events together.

Metal enthusiasts may be tempted to dismiss warfare as a worn-out lyrical theme, but Piarevaracien justifies its choice by treating the topic with a profound understanding of the concept. Emerging from a country whose borders were carved by centuries of struggle, death, and resurgence, the band recognises the perpetuity of war. Such is confirmed by the following Nietzsche quote, which has been included in the album booklet: “Ye shall love peace as a means to new wars and the short peace more than the long.” This reflects the pre-modern interpretation of peace as an anomalous interruption in between wars.

Piarevaracien brings to the table compositions which perfectly reflect the belligerent topic material of the album. Even though the group made a name for itself in black metal, the genre is only sporadically present in this new album’s sound. While the work defies strict stylistic classification, it is closest to the folk metal label. However, rather than relying on annoyingly exuberant recorders and other fairy instruments, Piarevaracien’s folklore here manifests itself through vibrant riffing that leaps enthusiastically over the drum and bass work—a method that is comparable to Kawir‘s Isotheos album. This approach results in melodies that are as colourful as they are varied, with riffs never outstaying their welcome before the band dives into the next movement.

Piarevaracien’s compositional prowess should not go unmentioned. U pošukach pačatku tych šlachoŭ incorporates a variety of influences ranging from black metal-style tremolo picking to melodeath rhythm patterns, and even vocals that dance on the stylistic border of hardcore. In the hands of less capable musicians, such a list of ingredients could easily have led to an obnoxious concoction, but Piarevaracien takes these divisive elements and weaves them into a well-rounded, tasteful plexus. Each composition offers smooth transitions between melodies and movements. Consequently, the album’s quality shines through both in its individual songs and on the album as a whole. It is largely this combination of individual and collective quality that makes U pošukach pačatku tych šlachoŭ such a joy to listen to. This impression is elevated further by the excellent production, which gives each of the instruments a place to shine. All of this makes the record’s slight blemishes—largely limited to occasionally off-key clean vocals—all the easier to forgive.

Due to the frightening consistency with which Piarevaracien establishes its mastery of tasteful folk metal, U pošukach pačatku tych šlachoŭ is one of 2016’s premier releases in the metal genre as a whole. With songs that are composed as well as they are played, this is an album with many highlights and virtually no moments of true weakness. Thematically, Piarevaracien’s handling of the topic of war goes way beyond the primal ‘blood ‘n’ gore’ fixation of more simple-minded metal formations. Instead of delivering hollow, misanthropic panegyrics to death, U pošukach pačatku tych šlachoŭ highlights the eternal return of a timeless warrior soul. By showing how young men who die in the struggle for a higher goal are forever hailed by their offspring, one can wonder if the looming end of Europe’s longstanding peace is truly something to be feared, or to be welcomed as an overdue liberation from the chains of meaninglessness.

[1] У пошуках пачатку тух шляхоў roughly translates as ‘Searching for the Beginning of Those Ways’.

Track List:

01) Razdziel I
02) Razdziel II
03) Razdziel III
04) Razdziel IV
05) Razdziel V
06) Razdziel VI
07) Razdziel VII

Written by: Degtyarov
Label: Crush the Desert (France) / CTD008 / Digipak CD
Folk Metal / Pagan Metal


He also served alongside the future emperor Majorian. Aegidius was either a founding member of Majorian and Ricimer's faction, or else he quickly joined it. After Majorian became Western Roman Emperor, Aegidius was granted the title of magister militum per Gallias (master of the soldiers for Gaul) in 458, as a reward for his loyalty. In the same year, Aegidius led troops at the Battle of Arelate, against the Visigoths under King Theodoric II. Aegidius is credited by ancient sources as being the primary cause for Theodoric II's defeat.

Theodoric II

Theoderic IITheodoricTheodoric II of the Ostrogoths

Western Roman Empire

Western EmpireWesternWest

Kingdom of the Suebi

SuebiSuebic KingdomGalicia
Theodoric continued his war on the Suevi for three months, but in April 459 he returned to Gaul, alarmed by the political and military movements of the new emperor, Majorian, and of the magister militum Ricimer—a half-Sueve, maybe a kinsman of Rechiar —while his allies and the rest of the Goths sacked Astorga, Palencia and other places, on their way back to the Pyrenees. When the Visigoths disposed of Rechiar, the royal bloodline of Hermeric vanished and the conventional mechanism for Suevi leadership died with it. In 456, one Aioulf took over the leadership of the Sueves.

Flavius Aetius

In 445 Majorian defeated a Frankish siege of Turonum, which was followed by a Frankish attack under Clodio in the region of Atrebatum, in Belgica Secunda. The foederati were stopped in an ambush near Vicus Helena, where Aetius directed the operations while his commander Majorian (later Emperor) fought with the cavalry. However, by 450 Aetius had already returned to good terms with the Franks. In 449 Chlodio died, and the patricius supported his younger son Merovaeus's claim to the throne. Aetius adopted him as his own son and sent him from Rome, where he had been an ambassador, to the Frankish court with many presents.

Nepotianus (magister militiae)

In 458 he was comes et magister utriusque militiae and together with Aegidius, he commanded the army of the Western Emperor, Majorian. That same year Majorian started a military campaign to re-conquer Gaul; the army, reinforced by some barbarian mercenaries, dislodged the Visigoths of Theodoric II from Arelate and obliged them to return to their condition of foederati. With the help of his new foederati, Majorian entered in the Rhone Valley, conquering its populations "some by arms and some by diplomacy". He defeated the Burgundians and besieged and conquered the city of Lugdunum: the rebel city was heavily fined, while the Bagaudae were forced to join the Empire.

Roman Gaul

Roman Gaul refers to Gaul under provincial rule in the Roman Empire from the 1st century BC to the 5th century AD.

The Visigoths (Visigothi, Wisigothi, Vesi, Visi, Wesi, Wisi; Westgoten Visigoti Visigots Visigodos Visigodos) were the western branches of the nomadic tribes of Germanic peoples referred to collectively as the Goths. These tribes flourished and spread throughout the late Roman Empire in Late Antiquity, or what is known as the Migration Period. The Visigoths emerged from earlier Gothic groups (possibly the Thervingi) who had invaded the Roman Empire beginning in 376 and had defeated the Romans at the Battle of Adrianople in 378. Relations between the Romans and the Visigoths were variable, alternately warring with one another and making treaties when convenient.

GothiaMarch of GothiaGothic March

ArelateArles FranceArles, France
Arles (Provençal Arle in both classical and Mistralian norms; Arelate in Classical Latin) is a city and commune in the south of France, in the Bouches-du-Rhône department, of which it is a subprefecture, in the former province of Provence.

Kingdom of the Burgundians

The Kingdom of the Burgundians or First Kingdom of Burgundy was established by Germanic Burgundians in the Rhineland and then in Savoy in the 5th century.

Visigothic Kingdom

Roman emperor

EmperoremperorsEmperor of the Roman Empire

RhoneRhone valleyRhône Valley
The Rhône (, ; Rhone ; ; Rodano ; Rôno ; Ròse ) is one of the major rivers of Europe and has twice the average discharge of the Loire (which is the longest French river), rising in the Rhône Glacier in the Swiss Alps at the far eastern end of the Swiss canton of Valais, passing through Lake Geneva and running through southeastern France. At Arles, near its mouth on the Mediterranean Sea, the river divides into two branches, known as the Great Rhône (le Grand Rhône) and the Little Rhône (le Petit Rhône). The resulting delta constitutes the Camargue region.

year 458 AD
Emperor Majorian builds a Roman fleet at Miseno and Ravenna. He strengthens the army, by recruiting a large number of barbarian mercenaries (Bastarnae, Burgundians, Huns, Ostrogoths, Rugii, Scythians and Suebi). Summer – The Vandals land in Campania, at the mouth of the Liri or the Garigliano River, and devastate the region. Majorian personally leads the Roman army and defeats the invaders near Sinuessa, destroying their ships on the seashore, loaded with booty. Battle of Arelate: Majorian defeats the Visigoths under King Theodoric II at Arles (Southern Gaul), near the Rhone River.

Germanic Wars

Roman-Germanic WarsGermanic invasionsBarbarian invasions
leads the Roman army to a victory over the Vandals near Sinuessa, Roman victory over the Visigoths in southern Gaul in the Battle of Arelate. 459, Seizure of Trier by Franks, Roman reconquest of southern Gaul and most of Hispania under Emperor Majorian. 460, Roman victory over the Suebi at Lucus Augusti, Roman fleet is destroyed by traitors paid by the Vandals, Attack on the kingdom of the Vandals cancelled. 461, Seventeen Vandal ships destroy forty Roman ships in a surprise attack. 463, Battle of Orleans. 465, Ostrogothic King Valamir dies in battle. 468, Invasion of the Vandal Kingdom by the Byzantine Empire, Defeat of the Byzantine Empire by the Vandals in the Battle of Cape Bon. 469, Ostrogoths

Gothic Wars

(Visigoths under Theodoric II against the Western Roman Empire under Majorian) In late 458 Majorian entered Septimania (now southern France) to attack Theodoric and reclaim the province for the empire. Majorian defeated Theodoric at the Battle of Arelate, forcing him to abandon Septimania and withdraw west to Aquitania. Under the new treaty with the Romans, the Visigoths had to relinquish their recent conquests in Hispania and return to federate status.

Fall of the Western Roman Empire

decline of the Roman Empirefall of the Roman Empirefall of Rome

Flavius Ricimer
Majorian led his field army north to fight the Alamanni, defeating them. Majorian was proclaimed emperor by his troops in a place called ad Columellas on April 1, 457. Realizing Majorian's potential as a puppet, Ricimer induced Leo to give his consent to this arrangement. Though Ricimer had expected to control his friend, Majorian proved to be a capable ruler and soon distanced himself from his magister militum. Majorian demonstrated his military skill through his re-conquest of Gaul and his campaigns in Hispania. Majorian's campaigns effectively subdued the Visigoths and returned them to their pre-Avitus foederati status, greatly increasing his standing among the senate and army.

EparchiusEparchius AvitusM. Maecilius Eparchius Avitus Augustus

Galla PlacidiaGalla Placidia Valentiniana Minor
Originally Emperor Valentinian intended to marry Placidia to a young man named Majorian (the future emperor), whom Oost describes as having "distinguished himself in a subaltern capacity fighting in Gaul against the Franks under Aëtius' own command." Doing so, according to Roman customs, would instantly link Majorian to the Imperial family and put him in line to succeed Valentinian. Once Flavius Aetius learned of this plan, he rusticated Majorian to his estates at some date before 451, and he was recalled to Rome only after Aetius' death.

Sidonius Apollinaris

SidoniusGaius Sollius Modestus Apollinaris SidoniusGaius Sollius Apollinaris Sidonius


Was planning on adding a spiral tube slide from the back, climbing ladder to the slider side, and a “sound machine” so the kids could pretend to be driving.

The neighbors and kids thought it was amazing. Sadly the village did not, even after trying to get an answer on doing this (to no avail, so we went for it), and ended up making me remove it.


A new method to implement comminution functions into DEM simulation of a size reduction system due to particle-wall collisions Academic Article uri icon

  • The ability to design a size reduction system prior to full scale experiments and to optimize existing systems has long been a goal of designers. Such a design and optimization could be achieved by correctly simulating any system under any operating condition. In this paper we present a new and innovative procedure to implement empirical comminution functions into DEM–CFD simulations. The paper is focused on the implementation procedures and not the DEM/CFD simulations, which deserve full attention. Therefore, this paper is not aimed to study any specific mill. The comminution functions include: initial strength distribution, selection function, breakage function and fatigue function. First, the traditional comminution functions (strength distribution, selection and breakage functions) and the recently investigated fatigue function are briefly described and modified. Then a procedure for implementing the functions into a DEM–CFD model or any other source to provide impact velocities and number of impacts, is described in detail. The implementation involves converting the probability comminution functions into individual particle properties by a random method and then converting the velocity dependent comminution functions into strength dependent ones. In this way, and mainly owing to the use of the fatigue function (which defines the weakening of those particles that are not breaking), a real size reduction system, in which each particle is subjected to multiple impacts at various velocities can be simulated. Three case studies for multiple impact conditions at the same average velocity (several impacts at the same velocity, various velocities at each impact and randomly selected velocities) are presented and analyzed in order to confirm qualitatively the procedure, although the comminution functions need to be further quantitatively modified. It should be emphasized that although the new procedure presents a step towards the final goal, some limitations do exist and some questions remain open.

publication date

  • January 1, 2009


The Distinguished Characteristics of Beowulf vs. Grendel

The Distinguished Characteristics of Beowulf vs. Grendel

?Jongwon Josh Lee Dr. Weibly English IV 4 November, 2012 The Distinguished Characteristics of Beowulf vs. Grendel In Beowulf, the hero defeats the “corpse-maker”, Grendel. Of Beowulf’s three battles, the fight with Grendel has the most significance. Both Grendel and Beowulf emerge unexpectedly, foreigners who respectively destroy and restore to make dramatic changes to the peace of the Spear-Danes. Beowulf also proves his heroic nature in terms of personality, such as his loyalty and love for fairness.

Furthermore, this battle is metaphorically significant, because it can be categorized as a battle between a rebel and a valiant protector. Beowulf emerges as the loyal stranger and the foreign protector of the land, who stops the rebellious destroyer; Grendel. The sudden appearances of Grendel and Beowulf, which accentuate disastrousness and heroism respectively, show the significance of this particular battle. Under the good, kind Hrothgar, the Spear-Danes enjoy peace for a long time, feasting and drinking in the magnificent Heorot, until a brutal monster emerges startlingly.

As the narrator observes, Grendel “… grabbed thirty men from their resting places and rushed to his lair, flushed up and inflamed from the raid, blundering back with the butchered corpses. ” (122-125). The monster arrives at Heorot in order to break the peace that he hates out of Jealousy. Hrothgar and his people are powerless against Grendel and lose hope until a warrior from Geatland comes to defeat the beast. This foreign warrior, who came to help unexpectedly, is Beowulf, the one who saves the Danes from Grendel. The purpose of his arrival is to help them out of such atrocity caused by Grendel.

Beowulf and his thanes “…vaulted over the side, out on to the sand, and moored their ship” (224-226). Grendel and Beowulf, both jump into the storyline unexpectedly. Grendel’s sudden appearance that breaks the long peace of the Danes emphasizes his wickedness, while Beowulf’s sudden rise and victory, that puts an end to the atrocity and the hopelessness the Danes had faced, accentuates the excitement of this battle. Moreover, the two show contrariness as well as in their purposes for the arrival at Heorot: Grendel comes to destroy, but Beowulf comes to help.

These emphases add to the significance of Beowulf’s fight against Grendel. The battle with Grendel is significant because it reveals Beowulf’s great personality: his loyalty and love for fairness. Grendel attacks the Spear-Danes, not Geatland, Beowulf’s homeland. Hrothgar greets Beowulf, mentioning the favor he had done for the Geats, when “A crew of seamen who sailed for me once with a gift-cargo across to Geatland…”(377-378). Beowulf crosses the sea to defeat Grendel, in order to repay the favor that had been provided to his father, Ecgtheow.

A warrior may be brave, but no warrior who is merely brave would do such an act. Beowulf’s loyalty shows is one the great traits in his personality. Grendel is a powerful monster, but does not use weapons. Acknowledging that Grendel fights with his bare hands, Beowulf refuses to use any weapons or armor, thinking that unfairness would occur if he uses his armaments. Beowulf boldly states, “No weapons, therefore, for either this night… ” (683-684). He shows his love for fairness, which is another one of his personality traits as a hero.

Such trait is underscored especially in the battle with Grendel, giving this battle even more significance. Beowulf fighting Grendel is metaphorically significant. Grendel had been expelled from the society to the swamp. Grendel dwelt in pain “…for a time in misery among the banished monsters, Cain’s clan, whom the Creator had outlawed and condemned as outcasts. ” (104-107). Due to the banishment and hatred he receives, Grendel feels defiance, which is the cause of his raid on Heorot: he could not stand the happiness going on in the mead hall.

Hereby, Grendel is an unstoppable rebel against a peaceful society, trying to overthrow the government and let anarchy take over. However, despite Grendel’s fierceness and power that the Spear Danes could not even dare to stop, Beowulf stops the rebellion, and protects the people and regains their peace. He is the heroic protector of the people who “was granted the glory of winning” (816-817) that overpowers Grendel who “was driven under the fen-banks, fatally hurt” (818-819). Such metaphors that can be symbolically used to describe the conflict between the two also increase the significance of this battle.

Beowulf gets involved in three intense battles, and all three battles hold specific significant factors of an epic poem. However the battle with Grendel has the most significance. The battle consists of the unexpected appearances that stress the disastrousness of the invasion and the heroism of Beowulf, the respectable personality traits shown through Beowulf’s actions, and the metaphoric significance of the two powers. Such traits indicate that the clash between Beowulf and Grendel is the paramount of the entire poem.


Solid Waste Cleanup Program Weights and Volumes for Project Estimates
Description of MaterialsApproximate Pounds/Cubic YardRemarks
Burn Dump Debris/Ash800-1000
Dry Loose
Wet for Dust Suppression
Wet mixed with soil
Construction Debris, Asphalt or Concrete: Loose2400
Construction Debris, Wood ; Uncompacted 400Increase up to 100% if compacted using heavy equipment
Loose/Dry. Plus 30% when compacted.
Gravel or Crushed Stone Loose/Dry2600Increase 20% if wet
Household Trash800
Liquid Waste1600202 gal./cubic yard ~ 7 Lbs./Gal.
E.g. Antifreeze, Waste Oil, Solvent
Metals, Un-compacted600e.g. Appliances, Metal Siding
Sand, Loose/Dry2400Increase 20% if damp and 30% if wet/compacted
Stone, Graded 8” max. Loose2700e.g. Gabion Construction. Increase 10% consolidated in place
Tire Burn Ash500-800
Tires, Auto and Pickup220Average 10 tires per cubic yard
Tires, OTRSee RemarksAverage 500 pounds per tire
Tires, Truck480Average 4 tires per cubic yard
Vehicles, Auto and PickupSee RemarksUse 3000 Pounds/Vehicle
Wood Chips, Shredded/Dry Wood Chips/Bark w/30% Soil300
Yard Waste (Vegetation) Loose600

Determination of Weights and Volumes of Onsite Materials

Pile volume can best be estimated by determining the area of the base and then multiplying by the average height of the pile. In many cases the base of a pile will resemble a rectangle where area is length times width (L x W). In other cases the pile may more closely resemble a triangle or other polygon. Use the appropriate geometry to calculate the base area. For average height, this usually must be estimated since often it is not prudent to climb a pile to get more exact height measurements. The height may be estimated by using a known reference (e.g., fellow inspector) for reference. Cubic yards can be determined by dividing cubic feet by 27. Depending upon the accuracy of the assumed measurements, the estimated volume could be within 10-15 percent of the actual volume.

The weight (tonnage) of a pile is determined by multiplying the volume by the density. CalRecycle’s Solid Waste Cleanup Program has developed approximate pounds per cubic yard (lbs/cu yd) estimates for various materials. The actual density depends on the homogeneous nature (uniformity) of the pile in both void space and material type. Unless the entire pile can be visualized, it will be difficult to determine an accurate tonnage estimate. Please note that density values in the table are general (rough) estimates only and the actual density could be up to (or exceed) a factor of three (either larger or smaller) depending upon the actual density of the material.

Determination of maximum weights and volumes that can be received:

Tons permitted to be received per day x 30 days = Maximum amount on site at any one time

Helpful formulas:

___ feet high X ___ feet wide X ___ feet long = ___ cubic feet/27 cubic feet per cubic yard = ___ cubic yards

___ cubic yards X 27 cubic feet per cubic yard = ___ cubic feet = height X width X length

The pile is 20 feet high X 40 feet wide X 253.1 feet long. This equates to about 202,479 cubic feet/27 cubic feet per cubic yard = approximately 7500 cubic yards.

___ cubic yards X ___ pounds per cubic yard (waste conversion factor) = ____ pounds/2000 pounds per ton = ____ tons

___ tons X 2000 pounds per ton/pounds per cubic yard = ___ cubic yards X 27 cubic feet per cubic yard = height X width X length

7500 cubic yards of wood X 400 pounds per yard (unchipped wood debris) = 3,000,000 pounds/2000 pounds per ton = 1500 tons


Pro-lifers call victory after UN shelves 'clone & kill' Belgian proposal

Contrary to earlier press reports released by most major media Friday, pro-life organizations have cried victory after the United Nations shelved competing proposals for a treaty to ban human cloning. The UN agreed to meet to discuss the issue again in February during the current session of the Sixth Committee in order to finalize a compromise proposal put forward by the Italian delegation.

The compromise means that the UN General Assembly will not vote on rival resolutions as erroneously stated in earlier press reports, claiming that the February discussions would lead to a less powerful declaration.

The Italian compromise proposal calls for countries to prohibit any attempts at the creation of human life through cloning and any research intended to achieve that aim and to ensure that in the application of life sciences human dignity shall be respected in all circumstances.

 In particular, the compromise states that women shall not be exploited (millions of eggs would be needed in order to carry out the research).

The compromise, however, does not preclude any subsequent consideration of a Convention.

The Costa Rican Resolution calling for a convention to Ban all human cloning would have no legal effect for several years due to the long process of negotiations and ratification and would only legally bind those countries ratifying it.   Thus, the immediate effect of passage of both the Costa Rican proposal for a total ban and the Italian Declaration would be to send a strong message that the world community wants countries to pass legislation banning all forms of human cloning.

A key pro-life insider at the United Nations explained to Catholic News Agency that "the reason our side did not pursue voting on the Costa Rican proposal (although we had over 60 co-sponsors and many more who promised to vote for it; and the other side had about 20 co-sponsors and we thought we could win a vote on the substance) was because the other side threatened a procedural 'No Action' motion that would delay action until next year's meeting of … the General Assembly in September."

The source revealed that many small countries - although almost all are for a total ban- are under enormous pressure from the opposition. According to the UN insider, there are only a small handful of countries that do not support a total ban on human cloning.

"The most important result of our over three years of work is that we have prevented the passage of a Convention that would approve of 'therapeutic cloning,' which appeared to be on a fast track at the beginning because of the total lack of knowledge and misinformation," the source said.

The side supporting a total ban on human cloning has a strong, legally consistent message. It claims that if the UN condones therapeutic cloning it would be the first time the international organization condones the creation of human life for the express purpose of using it for experimentation, a process that necessitates killing human beings in their embryonic stage for their stem cells.

In addition, for the first time, the UN would not only approve this, but would also require states to pass and enforce laws requiring that it be carried out.

Pro-lifers argue that the UN has never condoned or required the violation of the basic human rights of another human being, even though some countries were engaged in such violations.

The UN, therefore, has a responsibility to set ethical standards for the protection of the lives, dignity and worth of all human beings, even if some countries choose to violate these rights, pro-lifers say.

Costa Rica's ambassador to the UN, Bruno Stagno Ugarte, suggested that the United States and others would not be willing to bend on the words, "human life." He underscored that there was nothing in the Italian resolution on the issue that would prevent his country from reviving calls for an international treaty.

According to an AP story, the Belgians (sponsors of the resolution allowing research cloning) object to using "human life" because they fear it could be interpreted to ban all forms of human cloning. They would rather have a document that uses the language "human being." "This is the best compromise you are going to find," Stagno Ugarte told The Associated Press. "It's an important word, I can see that. But it's either that or it's 'embryo.'"

U.S. State Department spokesman Adam Ereli told the AP that the United States was still pleased with Friday's vote.

"It's our longstanding position that all human cloning is wrong, and we are proud of our efforts to prevent human cloning," he said. "So the fact that there isn't any action by the UN to endorse cloning is a moderate success."

"This, in effect, sends a very strong message: this is the direction in which the UN wants to go," said Jeanne E. Head, vice-president for international affairs for National Right to Life, commenting on the Italian resolution.


The Agreement is currently non-exclusive; however, Archer and the Partner intend to progress to an exclusive Collaboration Agreement in the near term. Due to the non-exclusive nature of the MTA, the company name of the Partner has not been stated.

The Agreement aims to contribute towards fabricating a proof-of-concept biosensor, comprising printable components capable of detecting disease state markers, such as antibodies or antigens. The Partner specializes in commercial biological detection technology and materials, and is concurrently developing and improving biosensing technology for emerging markets.

As the next steps in progressing the MTA, Archer will work with the Partner to identify technical and commercial avenues that address technology gaps in the biosensor market and are able to be serviced by low cost and multi-modal biosensing devices. A study is to be performed using in vitro diagnostic products available to the Partner for infectious disease serology, that aid in the diagnosis of viral, bacterial, parasitic and fungal diseases, while jointly assessing and optimizing the graphene and graphene ink biosensor componentry performance.

The collaboration with The University of Adelaide ARC Graphene Hub will continue with the aim of contributing to the development of a functional in vitro electrochemical carbon-based biosensor. The value-add development involves Archer’s direct access to infrastructure and personnel to prepare graphene-based materials, including functional derivatives like inks, needed for processing complex biosensing componentry and characterizing these materials to determine biocompatibilities that lead to molecular sensing.


Measurement System Basics

What is Fit?

One reason to set a tolerance is that the dimensional difference must be determined when engaging multiple parts, such as a shaft being engaged to a bore. This is called fit, or clearance.
When considering the fit, the way to approach measurement differs depending on whether to base measurement on the shaft or the hole. For example, when considering the diameter of the shaft as the basis, if the shaft only needs to go through the bore, a clearance fit can be used. If the shaft needs to be fixed to the bore, it must be an interference fit. For something in between, a location/transition fit is used.


San Diego police won’t act as immigration officers, mayor says

Mayors pledge to work together, on the steps of the historic Casa de la Cultura in Tijuana.
San Diego Mayor Kevin Faulconer, on the left, and Tijuana Mayor Juan Manuel Gastelum, center, walk into the historic Casa de la Cultura in Tijuana on March 13.
(Alejandro Tamayo / The San Diego Union-Tribune)

In a March 15 memo to the City Council, Faulconer reaffirmed that the city’s police department will not take part in a provision of the Immigration and Nationality Act that allows the federal government to enter into agreements with state and local law enforcement agencies to enforce immigration law.

“Such an agreement would essentially task local law enforcement officers with doing the job of the federal government and places an unfair financial burden on local taxpayers,” Faulconer’s memo says.

The mayor says city police officers do not stop people exclusively to determine their immigration status, and they do not ask the immigration status of people who are victims or witnesses of crimes. If a person is suspected of committing a crime, however, San Diego’s police officers work with regional, state and federal authorities to hold them accountable regardless of their immigration status.

“Our police department is committed to protecting everyone in our community and bringing criminals to justice,” he said in the memo. “The stronger the partnerships are between law enforcement and residents, the more successful we will be at keeping our neighborhoods safe.”

The memo reinforces the mayor’s continued non-endorsement of the new administration’s immigration policies. His recent comments in reaction to tougher policies from President Trump have reiterated the benefits of legal immigration and the importance of the cross-border relationship between San Diego and Tijuana to the region’s economy, culture and social fabric.

Stewart writes for the San Diego Union-Tribune.

L.A. County sheriff opposes ‘sanctuary state’ bill, saying it would hinder law enforcement

This tiny California town’s battle over ‘sanctuary city’ status started long before Trump

L.A. leaders challenge legality of Trump’s order to punish ‘sanctuary cities’


50 Reputation

9 Badges

6 years, 313 days

MaplePrimes Activity

These are questions asked by STHence

MapleSim already developed a Template for Linearization.

In addition, without the linearization template,

I have another differential equations that I can't find the integral of its solution by applying above 2 methods. Maple runs more than 10 mins without results. These equations are attached in this comment.

I really appreciate all your help.

Hi all,

I try many ways to solve the below DAEs, for example stiff=true, method = rkf45_dae, method=rosenbrock..., but my DAEs can't be solved. In different ways, there are different errors, such as "Warning, cannot evaluate the solution past the initial point, problem may be complex, initially singular or improperly set up", ""

Would you please show me how to solve the below DAEs and plot their solutions?

I really appreciate your help.

Hi all,

I have a differential equation as follows:

x(t)^2+cos(x(t))+diff(x(t), t)+diff(x(t), t, t) = 100

x(0)=2, D(x)(0) = 3;

After using "dsolve" to solve this equation, the solution x(t) is obtained.

How can we obtain the following function?

g(k) = 0kx2(t)dt

How can we plot the function g(k...

1 2 3 Page 1 of 3