Data analysis topics (#708)

* Local commit

* First part of  'data analysis' topics

* Delete pngs

* Second blast of edits and pics

* Delete that copy button pic

* Work in plot bar graph

* Dastardly png
This commit is contained in:
Galen Nickel 2018-03-14 15:54:32 -07:00 committed by GitHub
parent ecc9449b02
commit dbb26a2b51
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
35 changed files with 549 additions and 111 deletions

View File

@ -331,7 +331,12 @@
## #other
* [Hardware](/device)
* [Data logging](/device/data-logging)
* [Data Analysis](/device/data-analysis)
* [Plotting with LEDs](/device/data-analysis/led-plotting)
* [Viewing Data](/device/data-analysis/viewing)
* [Writing Data](/device/data-analysis/writing)
* [Generating Data](/device/data-analysis/generating)
* [Analyzing Data](/device/data-analysis/analyze)
* [Error codes](/device/error-codes)
* [Foil circuits](/device/foil-circuits)
* [MES events](/device/mes-events)

View File

@ -0,0 +1,15 @@
# Data Analysis
In addition to learning programming, the @boardname@ is a fantastic tool for observing and measuring things that happen in the natural world. The sciences rely on methods of observation, information gathering, and analysis. Using the @boardname@ with MakeCode, you can use the sensors and inputs to measure and capture physical events that occur. Meausrement data can then be sent for recording by other devices like another @boardname@ or a personal computer.
![Data graph logo](/static/mb/device/data-analysis/data-analysis.jpg)
The Data Viewer in the MakeCode editor allows you to stream and log data from your programs. It will do this when you write data values using the **[serial](/reference/serial)** write functions. When you try your code in the simulator or connect with USB, each value you write is saved and collected as a log for you to analyze later if you want.
These topics describe how to analyze your data using MakeCode with the @boardname@:
* [Plotting with LEDs](./data-analysis/led-plotting)
* [Viewing your data](./data-analysis/viewing)
* [Writing data](./data-analysis/writing)
* [Generating data](./data-analysis/generating)
* [Analyze](./data-analysis/analyze)

View File

@ -0,0 +1,110 @@
# Analyze
When it recognizes that data in the console has a format it can show in a chart, the Data Viewer will display it along with any new values that arrive. It also creates multiple charts if it notices more than one value stream in the incoming data.
If you want to keep the data and work with it later, you can download it and save it to your computer or mobile device.
## Charting value streams
As an example, let's generate some data for two values. The first value `x` makes a line on the chart. The second value is the square of the first value `x**2`shifted by `5`. This second value will show a chart of a parabola.
```blocks
for (let x = 0; x <= 10; x++) {
serial.writeValue("x", x)
serial.writeValue("y", (x - 5)**2)
basic.pause(1000)
}
```
![Charts of line and parabola](/static/mb/device/data-analysis/line-parabola.jpg)
Since we used ``||serial:serial write value||`` for each value, there are two charts, one for the `x` stream and the other for the `y` stream. Also, the values shown in the console window are formatted as name value pairs:
```
x:0
y:25
x:1
y:16
...
```
## Charting by data lines
Instead of writing multiple values separately, we can combine them in an array and write them together at once. Using the last example, let's combine each line and parabola value into an array.
```blocks
for (let x = 0; x <= 10; x++) {
serial.writeNumbers([x, (x - 5) ** 2])
basic.pause(1000)
}
```
![One chart of both line and parabola](/static/mb/device/data-analysis/line-parabola2.jpg)
This time both values are shown on the same chart. We used ``||serial:serial write numbers||`` to send both values at the same time. Instead of being written as name value pairs, the values are combined into one line of comma separated values (CSV). The Data Viewer recognizes the values as one stream and puts multiple data lines on the same chart. The console shows the data in CSV format:
```
0,25
1,16
2,9
3,4
...
```
## Raw data
The data in the console window appears exacly like it's written by your program. The data your program writes is called "raw" data. The Data Viewer reads the data and "cooks" it by deciding how to display it in a way that's useful to you.
You can select and copy the raw data from the console and paste it into another program if you want to work with it outside of the MakeCode editor. Also, there is a copy button next to the console window that let's you easily copy the raw data with one click.
![Copy button for console data](/static/mb/device/data-analysis/copy-button.gif)
## Downloading your data
You can download your data to save it to view in another program. Click the **Download** button above the chart view.
![Download button highlighted](/static/mb/device/data-analysis/download-button.jpg)
The data is formatted again and saved in file with a name like:
``microbit-data-13-20-2018-14-18-15-0700.csv``
The filename contains the date and time when the data was downloaded. If you open the file in an editor you can see how the Data Viewer reformatted it from how it was in the console.
### Download format
The download file will contain your data as lines of CSV. The downloaded data from the first example will look something like this:
```
sep=
time (source1) x time (source1) y
0 0 0 25
1.018 1 1.026 16
2.049 2 2.045 9
3.062 3 3.059 4
4.064 4 4.074 1
5.078 5 5.078 0
6.115 6 6.112 1
7.097 7 7.112 4
8.131 8 8.128 9
9.145 9 9.145 16
10.148 10 10.16 25
```
The first line says what character is used as the value separator. Characters other than a comma can be used as separators. The Data Viewer choose to use a `TAB` character. The next line contains the headings for the values. These are the names of each value when name value pairs are written to the console. Also, a time value (timestamp) is included for each value. The timestamp is the amount of time since the start of the program when the value was written.
The remaining lines contain the data values and their timestamps. Each line has one occurance of each value. So, for the example data above, each new `x` and `y` value is put together on the same line.
### Loading into a spreadsheet
Each line of data is placed into a row in the spreadsheet. The separators tell the spreadsheet which column the values go into.
![CSV data in spreadsheet](/static/mb/device/data-analysis/spreadsheet-data.jpg)
### Analyze in the spreadsheet
Spreadsheets are powerful tools for analyzing and looking at relationships in your data. Most spreadsheet programs can do advanced charting too. Here's a chart of our example data. You'll notice that it looks similar to the chart displayed by the MakeCode editor.
![Spreadsheet with chart](/static/mb/device/data-analysis/spreadsheet-chart.jpg)
The spreadsheet knows how to take the headings from the download file and use them as labels for the individual data lines.

View File

@ -0,0 +1,111 @@
# Generating data
## Sensor values
Most of the data you want to record probably comes from values measured by the sensors. The sensor values are read using ``||input:Input||`` blocks. They return to your program the current value measured by the sensor.
```block
serial.writeValue("accelX", input.acceleration(Dimension.X))
```
```block
serial.writeValue("heading", input.compassHeading())
```
```block
serial.writeValue("light", input.lightLevel())
```
```block
serial.writeValue("forceX", input.magneticForce(Dimension.X))
```
## Pin data
External sensors and devices connected to the board are read using the ``||pins:Pins||`` blocks. Here are some examples of reading the pins and reporting measurements from devices:
### Soil moisture
```block
let moisture = pins.analogReadPin(AnalogPin.P0)
serial.writeValue("moisture", moisture)
```
### Closed door detector
```block
pins.onPulsed(DigitalPin.P0, PulseValue.Low, () => {
serial.writeValue("DoorClosed", 1)
})
```
### I2C humidity sensor
```block
let humidity = pins.i2cReadNumber(61, NumberFormat.Int8LE)
serial.writeValue("humidity", humidity)
```
## Human events
Sometimes we want to track human interactions with a device. These events might be button presses, gestures, or pin touches.
### Example: Button tracker
Run this example in the editor and switch to the Data Viewer. Watch it plot your button presses like pulse events in the chart.
```blocks
let buttonValue = 0
basic.forever(() => {
if (input.buttonIsPressed(Button.A)) {
buttonValue = 1
} else {
buttonValue = 0;
}
serial.writeValue("ButtonA", buttonValue)
basic.pause(200)
})
```
Writing an additional value creates another stream that will appear in a separate chart. Adding and event for another button press, we can plot pulses on a second chart.
```blocks
let buttonValue = 0
basic.forever(() => {
if (input.buttonIsPressed(Button.A)) {
buttonValue = 1
} else {
buttonValue = 0;
}
serial.writeValue("ButtonA", buttonValue)
if (input.buttonIsPressed(Button.B)) {
buttonValue = 1
} else {
buttonValue = 0;
}
serial.writeValue("ButtonB", buttonValue)
basic.pause(200)
})
```
## Time and timestamps
A timestamp marks when you read a value or detect that an event happens. These are commonly written along with other data values as an additional data item. The data viewer will create timestamps for values it sees when you write them. If you are downloading and saving data from the Data Viewer, it creates the timestamps for you and you don't need to add them.
If you need your own time values, you can make them from the **running time** of the board.
```block
input.onGesture(Gesture.Shake, () => {
serial.writeValue("shaken", input.runningTime())
})
```
Also, you can write your own custom CSV with your own timestamp:
```blocks
serial.writeLine("timestamp,temp,light")
basic.forever(() => {
serial.writeNumbers([input.runningTime(), input.temperature(), input.lightLevel()])
basic.pause(30000)
})
```

View File

@ -0,0 +1,97 @@
# Plotting data with LEDs
To quickly see and record your data values, use the ``||led:plot bar graph||`` block. It will plot a value on the LED screen and write a number to console at the same time.
```blocks
basic.forever(() => {
let i = 0
while (i < 25) {
led.plotBarGraph(i, 25)
i += 2
basic.pause(1000)
}
})
```
Here's a plot of values from `1` to `25`:
```sim
basic.forever(() => {
let i = 0
while (i < 25) {
led.plotBarGraph(i, 25)
i += 2
basic.pause(1000)
}
})
```
## Plot range
A number of LEDs will light up to show how much the value is related to the _high_ number in the second arguement. The high number sets the _range_ of values to show. If the high number is set to `16` and you want to plot the value of `8`, then that value is half of the range.
```block
led.plotBarGraph(8, 16)
```
So, about half of the LEDs will light up to represent that number.
```sim
basic.forever(() => {
basic.clearScreen()
basic.showString("8 up to 16")
basic.clearScreen()
basic.pause(500)
led.plotBarGraph(8, 16)
basic.pause(2000)
})
```
## Autoranging
If you don't know what the largest number to plot will be, then you can set the high number to `0`. This lets the largest value plotted be the range value. This is called _autoranging_. To see how it works, make an array of numbers and plot them. Put a large value in the middle of the array with smaller values before and after it. Watch as the same values are plotted with fewer lights after the largest number is plotted.
```blocks
let values = [4,8,12,16,50,4,8,12,16]
for (let i = 0; i < values.length; i++) {
led.plotBarGraph(values[i], 0)
basic.pause(1000)
}
```
## Recording plotted values
The ``||led:plot bar graph||`` also sends the number value it's plotting to the console. You can see the output in the Data Viewer. It charts the values and they appear as indivdual numbers in console.
```blocks
input.onButtonPressed(Button.B, () => {
for (let i = 0; i < 25; i++) {
if (i % 2 > 0) {
led.plotBarGraph(0, 0)
} else {
led.plotBarGraph(i, 24)
}
basic.pause(500)
}
})
```
The chart for the the plotted values from the example above:
![Plot LEDs chart result](/static/mb/device/data-analysis/plot-bar-graph.jpg)
The values appear in the console as just one number per line:
```
0
2
0
4
0
6
...
```
## See also
[plot bar graph](/reference/led/plot-bar-graph)

View File

@ -0,0 +1,50 @@
# Viewing your data
## Recording data
When your code is writing data, and the editor is recording it, the **Show data** button is displayed in the simulator under the board and simulation controls.
![Show data button](/static/mb/device/data-analysis/show-data.jpg)
## Data view window
If you press the **Show data** button, the editor will switch from the **Blocks** or **JavaScript** view to display a charting window and a text console.
![Data chart and console](/static/mb/device/data-analysis/data-view.jpg)
The console window under the graph will show the data in the format it was written. The chart is the visual representation of the values appearing on the console.
## Time scroll
The chart will display new data as it arrives. The chart will scroll with time to continue to display new values.
![Data chart time scroll](/static/mb/device/data-analysis/time-scroll.gif)
## Data view controls
The chart window shows the data view controls on top of the chart.
![Data view controls](/static/mb/device/data-analysis/data-view-controls.jpg)
Here's what the controls do:
**Return**: The return button switches the view back to previous code window (either Blocks or JavaScript).
![Return button](/static/mb/device/data-analysis/return-button.jpg)
**Source**: Tells you where the data is coming from. If the code writing the data is running in the simulator, then the source is **Simulator**. If your code is running on the @boardname@ and connected by USB, the source is **@boardname@**.
![Source label](/static/mb/device/data-analysis/source-label.jpg)
**Pause**: The pause button will stop the display of new values and stop scrolling. When you resume, the chart starts again with the current value written.
![Pause button](/static/mb/device/data-analysis/pause-button.jpg)
**Resume**: The resume button will start displaying new values after the **Pause** button was pressed.
![Resume button](/static/mb/device/data-analysis/resume-button.jpg)
**Download**: The download button collects the data your code has written and downloads it to your computer as a file called something like _data-11-2018-23-00-0700.csv_. The numbers in the filename are the date and time when the file is created. The file may automatically open in an editor or spreadsheet if one of those programs is associated with _csv_ files.
![Download button](/static/mb/device/data-analysis/download-button.jpg)

View File

@ -0,0 +1,160 @@
# Writing data
While you're using MakeCode, all data written by the [serial](/reference/serial) functions is recorded by the MakeCode editor. This happens when you try your code in the simulator and also when the @boardname@ is connected to a computer running the MakeCode app with USB.
## Data formats
The Data Viewer recognizes the format of your output and decides how to display it. If your data is a _stream_ of values it will plot them in the chart window and show them as text in the console. If your data is just text information, it appears only in the console window.
You can write data in these format types:
* Text ([string](/types/string)): `"Hello there!"`
* [Numbers](/types/number): `"354"`
* [Number arrays](/types/array): `"11,3,45,76"`
* Name value pairs: `"accelY:956"`
## Text output
Text is formed by simply using string data. The text data is not recorded in the console until a complete "line" of text is written. A line is just a string with some special characters (ones you don't actually see printed) at the end. You can write several strings, say:
```block
serial.writeString("Hello ");
serial.writeString("from ")
serial.writeString("micro:bit")
```
This text, though, won't appear in the editor console until it becomes a complete line. If you follow the string writes with a line in your code, then the line will show up:
```block
serial.writeString("Hello ")
serial.writeString("from ")
serial.writeString("micro:bit")
serial.writeLine("")
```
Text output is used mostly for messages since number formats have meaning when used for data analysis.
The blank ``||serial:serial write line||`` adds the special line ending characters and turns the previous strings into a complete line of text. The whole line could simply be written out with just one function:
```block
serial.writeLine("Hello from micro:bit")
```
When you're writing only text, it appears only in the console view and not in the chart window. This example writes four messages of text:
```block
for (let i = 0; i < 4; i++) {
serial.writeLine("Hello from micro:bit " + i)
}
```
The four messages appear in the console window:
![Console output](/static/mb/device/data-analysis/console-output.jpg)
## Writing numbers
In order for the Data Viewer to recognize a number as value and show in the chart, it has to be in a line by itself. So, numbers written individually won't be charted:
```block
serial.writeNumber(1)
serial.writeNumber(2)
serial.writeNumber(3)
```
The numbers don't show up as single values because they appear in the output as a string: `"123"`. Also, the string doesn't form a complete line so it doesn't show up in the console window either. You could add a blank line to the numbers alreay written. If you did this, you would have just one value charted which is `123`:
```block
serial.writeNumber(1)
serial.writeNumber(2)
serial.writeNumber(3)
serial.writeLine("")
```
Here's a way to chart the numbers individually:
```block
for (let i = 0; i < 4; i++) {
serial.writeNumber(i)
serial.writeLine("")
}
```
It's much better though to use a value stream by writing the numbers as [name value pairs](#name-value-pairs).
## Number arrays
Numbers in arrays are displayed on separate data lines on the same chart. You can use ``||serial:serial write numbers||`` to write several values at once. The numbers are written to the output as _comma separated values_ (CSV). The array of numbers:
```block
let values = [0,1,2,3,4];
```
is written to the output in the form of a CSV string as: `"0,1,2,3,4"`.
The Data Viewer recognizes this as an array of numbers and charts them on separate data lines:
```block
let values = [0, 1, 2, 3, 4]
basic.forever(() => {
serial.writeNumbers(values)
})
```
Data lines are shown for each value in the array:
![Data lines on chart](/static/mb/device/data-analysis/data-lines.jpg)
Give this example a try and watch the chart make a diamond pattern from two triangle waves:
```blocks
let diamond: number[] = []
basic.forever(() => {
for (let i = 0; i <= 10 - 1; i++) {
if (i < 5) {
diamond[0] = i
diamond[1] = 5 - i
} else {
diamond[0] = 10 - i
diamond[1] = i - 5
}
serial.writeNumbers(diamond)
basic.pause(500)
}
})
```
It will look like this:
![Data pattern on chart](/static/mb/device/data-analysis/diamond-chart.jpg)
## Name value pairs
A very common way to report and record data is to use a _name value pair_ (NVP). A value is given a name so you know what it's for or where it came from. The name value pair is written with the format of _name:value_. There is a name, a colon character, and the value all together on one line. The line is written to the output and the Data Viewer recognizes it as a value and charts it as part of a value stream. The value stream is based on the name so any new values received with the same name are displayed on the chart that's showing values for that name. If more than one type of name value pair is found, the Data Viewer makes another value stream and shows those values on a different chart.
If you want to report the values for both temperature and light, you can make separate name value pairs for them. The name value pairs are written using ``||serial:serial write value||``. This function writes the values to the output as lines in a format like this:
```
temp:17
light:118
temp:18
light:117
```
```blocks
basic.forever(() => {
serial.writeValue("temp", input.temperature())
serial.writeValue("light", input.lightLevel())
basic.pause(5000)
})
```
Two charts display each value stream:
![Two charts for each value type](/static/mb/device/data-analysis/two-value-chart.jpg)
The console output shows the different name value pairs too:
![Two values in console output](/static/mb/device/data-analysis/two-value-console.jpg)

View File

@ -1,110 +0,0 @@
# Data logging
The simulator in the MakeCode editor allows you to stream and log data from your programs. It will do this when you write data values using the **[Serial](/reference/serial)** write functions. When you try your code in the simulator, each value you write is saved and collected as a log for you to analyze later if you want.
## Record some data
Let's write some values and see what the simulator records for us. Copy this code into the MakeCode editor and press the `A` button in the simulator.
```blocks
input.onButtonPressed(Button.A, () => {
for (let i = 0; i <= 10 - 1; i++) {
serial.writeValue("count", i * 10)
basic.pause(1000)
}
})
```
In the simulator, a logging view appears below the board to show the values your program writes. Also, there's a small chart next to it showing how your values are changing over time. Your data and chart have a color code that changes each time you run the program to show that a different data _stream_ is being logged. In this example, our data stream is colored red.
![Data logger](/static/mb/device/data-log.png)
### Plot values on the LEDs
To quickly see and log your data values, use the ``||led:plot bar graph||`` block. It will plot a value with the LEDs and write it to the data log too.
```blocks
input.onButtonPressed(Button.B, () => {
for (let i = 0; i <= 25 - 1; i++) {
if (i % 2 > 0) {
led.plotBarGraph(0, 0)
} else {
led.plotBarGraph(i, 24)
}
basic.pause(300)
}
})
```
A number of LEDs will light up to show how much the value is related to the _high_ number in the second arguement. The simualtor and the logging area show this when the loop index is `14` in the code example:
![Bar graph logger](/static/mb/device/bar-graph-log.png)
## Analyze the data
If you click the data logging area in the simulator, a window pops up called **Analyze Data**. Click on **Download data** to save the data file. If you have a spreadsheet program installed, it may open the data file automatically. If not, you can open it later yourself in your spreadsheet program.
![Analyze data window](/static/mb/device/analyze-data.png)
If you look at the contents of this data file in an editor, it will look like this:
```csv
red time, red count, log time, log source, log message
0, 0, 0, red-394085874039518505560.6935929915907719, count:90
1.003, 10
2.001, 20
2.999, 30
3.999, 40
4.998, 50
6.016, 60
7.002, 70
7.997, 80
8.998, 90
```
Each row in the file has different values with commas in between them. This gives the file a certain format called **Comma Separated Value**, or **CSV** as a short name. A row with three data values are written like: _item 1, item 2, item 3_. These three values make a row of three _columns_ in the data file. Sometimes, the first row of a CSV file is used say what data values in the columns mean. These are called column _headers_.
Spreadsheet programs know how to read CSV files and can put each data value into the proper row and column. Let's bring the file into a spreadsheet and see what it looks like.
![See data in spreadsheet](/static/mb/device/spreadsheet-data.png)
You can see that the value of the `count` variable is recorded ten times under the heading called `red count`. Just before it is another column called `red time`. The simulator automatically records the time when a value is written to the logging stream and places it on the same row. Since this stream is called the _red_ stream, the color name is added to the header names for both the recording time and our value.
The other columns in the data file give more information about the value we were streaming to the log.
* `log time`: The time in the logging stream when our value was first recorded.
* `log source`: A unique name to identify this stream in the log file.
* `log message`: A helper note about our value. It's usually full string of the last value.
### Charting
To get a chart of your data, you can just select the columns that contain your values and their recording times. Use the **scatter plot** charting feature of the spreadsheet program to display a nice graph of your data. Here you see the values increasing from `0` to `90` over nine seconds of time.
![Chart the data in spreadsheet](/static/mb/device/spreadsheet-chart.png)
### Multiple values
You can include more than one value in a log stream. Let's say you want to write values to show a parabola when the data is charted in the spreadsheet program. To record data points for the parabola, we write values for **x** from `-10` to `10`. For the cooresponding **y** values, we write `x * x` for each **x** value written.
```blocks
input.onButtonPressed(Button.A, () => {
for (let i = 0; i <= 21 - 1; i++) {
let x = i - 10
serial.writeValue("x", x)
serial.writeValue("y", x*x)
basic.pause(1000)
}
})
```
What you'll see is that the simulator inserted some more columns for the second value written. The data columns for our values are called `green x` and `green y`. There's a `green time` column now for both of the data values. Each value gets its recording time saved.
This time there are two row entries (one for each data value in the logging stream) added next to the data columns for `log time`, `log source`, and `log message`.
If you want to chart with just your data and leave the recording times out, select just the value columns.
![Multiple data values in spreadsheet](/static/mb/device/spreadsheet-multi.png)
## See also
[write value](/reference/serial/write-value), [uart-write-value](/reference/bluetooth/uart-write-value),
[plot bar graph](/reference/led/plot-bar-graph)

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 19 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 40 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 38 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 4.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 23 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 17 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 4.0 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 4.4 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.6 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 7.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 52 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 38 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 404 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 96 KiB