from

Operator

  from — source data from pools, files, or URIs

Synopsis

from <pool>[@<commitish>]
from <pattern>
file <path> [format <format>]
get <uri> [format <format>]
from (
   pool <pool>[@<commitish>] [ => <branch> ]
   pool <pattern>
   file <path> [format <format>] [ => <branch> ]
   get <uri> [format <format>] [ => <branch> ]
   pass
   ...
)

Description

The from operator identifies one or more data sources and transmits their data to its output. A data source can be

• the name of a data pool in a Zed lake, with optional commitish;
• the names of multiple data pools, expressed as a regular expression or glob pattern;
• a path to a file;
• an HTTP, HTTPS, or S3 URI; or
• the pass operator, to treat the upstream pipeline branch as a source.

Note

File paths and URIs may be followed by an optional format specifier.

Sourcing data from pools is only possible when querying a lake, such as via the zed command or Zed lake API. Sourcing data from files is only possible with the zq command.

When a single pool name is specified without @-referencing a commit or ID, or when using a pool pattern, the tip of the main branch of each pool is accessed.

In the first four forms, a single source is connected to a single output. In the fifth form, multiple sources are accessed in parallel and may be joined, combined, or merged.

A pipeline can be split with the fork operator as in

from PoolOne | fork (
  => op1 | op2 | ...
  => op1 | op2 | ...
) | merge ts | ...

Or multiple pools can be accessed and, for example, joined:

from (
  pool PoolOne => op1 | op2 | ...
  pool PoolTwo => op1 | op2 | ...
) | join on key=key | ...

Similarly, data can be routed to different pipeline branches with replication using the switch operator:

from ... | switch color (
  case "red" => op1 | op2 | ...
  case "blue" => op1 | op2 | ...
  default => op1 | op2 | ...
) | ...

Input Data

Examples below below assume the existence of the Zed lake created and populated by the following commands:

export ZED_LAKE=example
zed -q init
zed -q create -orderby flip:desc coinflips
echo '{flip:1,result:"heads"} {flip:2,result:"tails"}' |
  zed load -q -use coinflips -
zed branch -q -use coinflips trial 
echo '{flip:3,result:"heads"}' | zed load -q -use coinflips@trial -
zed -q create numbers
echo '{number:1,word:"one"} {number:2,word:"two"} {number:3,word:"three"}' |
  zed load -q -use numbers -
zed query -f text '
  from :branches
  | yield pool.name + "@" + branch.name
  | sort'

The lake then contains the two pools:

coinflips@main
coinflips@trial
numbers@main

The following file hello.zson is also used.

{greeting:"hello world!"}

Examples

Source structured data from a local file

zq -z 'file hello.zson | yield greeting'

=>

"hello world!"

Source data from a local file, but in line format

zq -z 'file hello.zson format line'

=>

"{greeting:\"hello world!\"}"

Source structured data from a URI

zq -z 'get https://raw.githubusercontent.com/brimdata/zui-insiders/main/package.json
       | yield productName'

=>

"Zui - Insiders"

Source data from the main branch of a pool

zed -lake example query -z 'from coinflips'

=>

{flip:2,result:"tails"}
{flip:1,result:"heads"}

Source data from a specific branch of a pool

zed -lake example query -z 'from coinflips@trial'

=>

{flip:3,result:"heads"}
{flip:2,result:"tails"}
{flip:1,result:"heads"}

Count the number of values in the main branch of all pools

zed -lake example query -f text 'from * | count()'

=>

5

Join the data from multiple pools

zed -lake example query -z '
  from coinflips | sort flip
  | join (
    from numbers | sort number
  ) on flip=number word'

=>

{flip:1,result:"heads",word:"one"}
{flip:2,result:"tails",word:"two"}

Use pass to combine our join output with data from yet another source

zed -lake example query -z '
  from coinflips | sort flip
  | join (
    from numbers | sort number
  ) on flip=number word
  | from (
    pass
    pool coinflips@trial =>
      c:=count()
      | yield f"There were {int64(c)} flips"
  ) | sort this'

=>

"There were 3 flips"
{flip:1,result:"heads",word:"one"}
{flip:2,result:"tails",word:"two"}